Eungyeup Kim
eungyeuk (at) cs (dot) cmu (dot) edu
I am a second-year PhD student in Computer Science Department at Carnegie Mellon University, advised by Prof. Zico Kolter.
My research interest lies in robust ML under distribution shifts and foundation models.
I did my Master's studies at Korea Advanced Institute of Science and Technology (KAIST) under Prof. Jaegul Choo.
Google Scholar
 / 
GitHub  / 
CV  / 
Twitter
|
|
* indicates equal contribution
Preprints
|
|
Reliable Test-Time Adaptation via Agreement-on-the-Line
Eungyeup Kim, Mingjie Sun
, Aditi Raghunathan, Zico Kolter
Neural Information Processing Systems (NeurIPS) 2023 Workshop on Distribution Shifts (DistShift).
abstract /
paper
Test-time adaptation (TTA) methods aim to improve robustness to distribution shifts by adapting models using unlabeled data from the shifted test distribution.
However, there remain unresolved challenges that undermine the reliability of TTA, which include difficulties in evaluating TTA performance, miscalibration after TTA, and unreliable hyperparameter tuning for adaptation.
In this work, we make a notable and surprising observation that TTAed models strongly show the agreement-on-the-line phenomenon (Baek et al., 2022) across a wide range of distribution shifts.
We find such linear trends occur consistently in a wide range of models adapted with various hyperparameters, and persist in distributions where the phenomenon fails to hold in vanilla models (i.e., before adaptation).
We leverage these observations to make TTA methods more reliable in three perspectives: (i) estimating OOD accuracy (without labeled data) to determine when TTA helps and when it hurts, (ii)
calibrating TTAed models without label information, and (iii) reliably determining hyperparameters for TTA without any labeled validation data.
Through extensive experiments, we demonstrate that various TTA methods can be precisely evaluated, both in terms of their improvements and degradations.
Moreover, our proposed methods on unsupervised calibration and hyperparameters tuning for TTA achieve results close to the ones assuming access to ground-truth labels, in terms of both OOD accuracy and calibration error.
|
|
Learning Debiased Representation via Disentangled Feature Augmentation
Jungsoo Lee*, Eungyeup Kim* Juyoung Lee, Jihyeon Lee, Jaegul Choo
Conference on Neural Information Processing Systems (NeurIPS), 2021 (Oral).
abstract /
paper /
code /
Image classification models tend to make decisions based on peripheral
attributes of data items that have strong correlation with a target variable (i.e., dataset bias).
These biased models suffer from the poor generalization capability when evaluated on unbiased datasets.
Existing approaches for debiasing often identify and emphasize those samples with no such correlation
(i.e., bias-conflicting) without defining the bias type in advance. However, such bias-conflicting
samples are significantly scarce in biased datasets, limiting the debiasing capability of these approaches.
This paper first presents an empirical analysis revealing that training with "diverse" bias-conflicting
samples beyond a given training set is crucial for debiasing as well as the generalization capability.
Based on this observation, we propose a novel feature-level data augmentation technique in order to
synthesize diverse bias-conflicting samples. To this end, our method learns the disentangled
representation of (1) the intrinsic attributes (i.e., those inherently defining a certain class) and
(2) bias attributes (i.e., peripheral attributes causing the bias), from a large number of bias-aligned
samples, the bias attributes of which have strong correlation with the target variable. Using the disentangled
representation, we synthesize bias-conflicting samples that contain the diverse intrinsic attributes of
bias-aligned samples by swapping their latent features. By utilizing these diversified bias-conflicting
features during the training, our approach achieves superior classification accuracy and debiasing
results against the existing baselines on both synthetic as well as a real-world dataset.
|
|
Deep Edge-Aware Interactive Colorization against Color Bleeding Effects
Eungyeup Kim*, Sanghyeon Lee*, Jeonghoon Park*, Somi Choi,
Choonghyun Seo, Jaegul Choo
IEEE International Conference on Computer Vision (ICCV), 2021 (Oral).
abstract /
paper /
project
Deep image colorization networks often suffer from the color-bleeding
artifact, a problematic color spreading near the boundaries between adjacent objects. The color-bleeding
artifacts debase the reality of generated outputs, limiting the applicability of colorization models
on a practical application. Although previous approaches have tackled this problem in an automatic
manner, they often generate imperfect outputs because their enhancements are available only in limited
cases, such as having a high contrast of gray-scale value in an input image. Instead, leveraging user
interactions would be a promising approach, since it can help the edge correction in the desired
regions. In this paper, we propose a novel edge-enhancing framework for the regions of interest, by
utilizing user scribbles that indicate where to enhance. Our method requires minimal user effort to
obtain satisfactory enhancements. Experimental results on various datasets demonstrate that our
interactive approach has outstanding performance in improving color-bleeding artifacts against the
existing baselines.
|
|
BiaSwap: Removing Dataset Bias with Bias-Tailored Swapping Augmentation
Eungyeyp Kim*, Jihyeon Lee*, Jaegul Choo
IEEE International Conference on Computer Vision (ICCV), 2021
abstract /
paper
Deep neural networks often make decisions based on
the spurious correlations inherent in the dataset, failing
to generalize in an unbiased data distribution. Although
previous approaches pre-define the type of dataset bias to
prevent the network from learning it, recognizing the bias
type in the real dataset is often prohibitive. This paper proposes a novel bias-tailored augmentation-based approach,
BiaSwap, for learning debiased representation without requiring supervision on the bias type. Assuming that the
bias corresponds to the easy-to-learn attributes, we sort the
training images based on how much a biased classifier can
exploits them as shortcut and divide them into bias-guiding
and bias-contrary samples in an unsupervised manner. Afterwards, we integrate the style-transferring module of the
image translation model with the class activation maps of
such biased classifier, which enables to primarily transfer the bias attributes learned by the classifier. Therefore,
given the pair of bias-guiding and bias-contrary, BiaSwap
generates the bias-swapped image which contains the bias
attributes from the bias-contrary images, while preserving
bias-irrelevant ones in the bias-guiding images. Given such
augmented images, BiaSwap demonstrates the superiority
in debiasing against the existing baselines over both synthetic and real-world datasets.
Even without careful supervision on the bias, BiaSwap achieves a remarkable performance on both unbiased and bias-guiding samples, implying the improved generalization capability of the model.
|
|
Reference-Based Sketch Image Colorization Using Augmented-Self Reference and Dense Semantic Correspondence
Junsoo Lee*, Eungyeup Kim*, Yunsung Lee, Dongjun Kim, Jaehyuk Chang, Jaegul Choo
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020
abstract /
paper /
project /
video
This paper tackles the automatic colorization task of a sketch image
given an already-colored reference image. Colorizing a sketch image is in high demand in comics, animation,
and other content creation applications, but it suffers from information scarcity of a sketch image.
To address this, a reference image can render the colorization process in a reliable and user-driven
manner. However, it is difficult to prepare for a training data set that has a sufficient amount of
semantically meaningful pairs of images as well as the ground truth for a colored image reflecting a
given reference (e.g., coloring a sketch of an originally blue car given a reference green car). To
tackle this challenge, we propose to utilize the identical image with geometric distortion as a virtual
reference, which makes it possible to secure the ground truth for a colored output image. Furthermore,
it naturally provides the ground truth for dense semantic correspondence, which we utilize in our
internal attention mechanism for color transfer from reference to sketch input. We demonstrate the
effectiveness of our approach in various types of sketch image colorization via quantitative as well
as qualitative evaluation against existing methods..
|
|