[[2212.05166] A systematic literature review of cyberwarfare and state-sponsored hacking teams](http://arxiv.org/abs/2212.05166) #secureIt is expected that the creation of next-generation wireless networks would result in the availability of high-speed and low-latency connectivity for every part of our life. As a result, it is important that the network is secure. The network's security environment has grown more complicated as a result of the growing number of devices and the diversity of services that 5G will provide. This is why it is important that the development of effective security solutions is carried out early. Our findings of this review have revealed the various directions that will be pursued in the development of next-generation wireless networks. Some of these include the use of Artificial Intelligence and Software Defined Mobile Networks. The threat environment for 5G networks, security weaknesses in the new technology paradigms that 5G will embrace, and provided solutions presented in the key studies in the field of 5G cyber security are all described in this systematic literature review for prospective researchers. Future research directions to protect wireless networks beyond 5G are also covered.
[[2212.05219] Specular: Towards Trust-minimized Blockchain Execution Scalability with EVM-native Fraud Proofs](http://arxiv.org/abs/2212.05219) #secureAn optimistic rollup (ORU) enables refereed delegation of computation from a blockchain (L1) to an untrusted remote system (L2), by allowing state updates posted on-chain to be disputed by any party via an interactive fraud proof (IFP) protocol. Existing systems that utilize this technique have demonstrated up to a 20x reduction in transaction fees. The most popular ORUs today, in active development, strive to extend existing Ethereum client software to support IFP construction, aiming to reuse prior L1 engineering efforts and replicate Ethereum Virtual Machine (EVM) semantics at L2. Unfortunately, to do so they tightly couple their on-chain IFP verifier with a specific client program binary--oblivious to its higher-level semantics. We argue that this approach (1) precludes the trust-minimized, permissionless participation of multiple Ethereum client programs, magnifying monoculture failure risk; (2) leads to an unnecessarily large and complex trusted computing base that is difficult to independently audit; and, (3) suffers from a frequently-triggered, yet opaque upgrade process--both further increasing auditing overhead and complicating on-chain access control. In this work, we aim to build a secure, trust-minimized ORU that addresses these problems, while preserving scalability and dispute resolution efficiency. To do so, we design an IFP system native to the EVM, that enforces Ethereum's specified semantics precisely at the level of a single EVM instruction. We present Specular, an ORU which leverages an off-the-shelf Ethereum client--modified minimally to support IFP construction--demonstrating the practicality of our approach.
[[2212.05462] Cryptanalysis and designing chaos-based irreversible and parallel key expansion module over Galois field](http://arxiv.org/abs/2212.05462) #secureFrom the security criteria of irreversibility, parallelizability and independence, we cryptanalyzed the key expansion modules of candidate block ciphers of AES, the results revealed that there exist some weaknesses inside, which may be explored by the attacker. Hence, we designed a more secure key expansion module that the round-key can satisfy three criteria above. First, we constructed a non-degenerate 2D chaotic map (2D-{\pi}eCM) with ergodicity in phase space and sufficient large chaotic range. Then based on 2D-{\pi}eCM and polynomial multiplication over Galois field, we designed an irreversible key expansion module, which could transform the initial key of arbitrary length to desired number of independent round keys in parallel. Security and statistical results demonstrated the flexible and effectiveness of the proposed key expansion module.
[[2212.05709] HOTCOLD Block: Fooling Thermal Infrared Detectors with a Novel Wearable Design](http://arxiv.org/abs/2212.05709) #securityAdversarial attacks on thermal infrared imaging expose the risk of related applications. Estimating the security of these systems is essential for safely deploying them in the real world. In many cases, realizing the attacks in the physical space requires elaborate special perturbations. These solutions are often \emph{impractical} and \emph{attention-grabbing}. To address the need for a physically practical and stealthy adversarial attack, we introduce \textsc{HotCold} Block, a novel physical attack for infrared detectors that hide persons utilizing the wearable Warming Paste and Cooling Paste. By attaching these readily available temperature-controlled materials to the body, \textsc{HotCold} Block evades human eyes efficiently. Moreover, unlike existing methods that build adversarial patches with complex texture and structure features, \textsc{HotCold} Block utilizes an SSP-oriented adversarial optimization algorithm that enables attacks with pure color blocks and explores the influence of size, shape, and position on attack performance. Extensive experimental results in both digital and physical environments demonstrate the performance of our proposed \textsc{HotCold} Block. \emph{Code is available: \textcolor{magenta}{https://github.com/weihui1308/HOTCOLDBlock}}.
[[2212.05099] A Systematic Literature Review on Smart Contracts Security](http://arxiv.org/abs/2212.05099) #securitySmart contracts are blockchain-based algorithms that execute when specific criteria are satisfied. They are often used to automate the implementation of an agreement so that all parties may be confident of the conclusion right away, without the need for an intermediary or additional delay. They can also automate a process so that the following action is executed when circumstances are satisfied. This study seeks to pinpoint the most significant weaknesses in smart contracts from the viewpoints of their internal workings and software security flaws. These are then addressed using various techniques and tools used across the industry. Additionally, we looked into the limitations of the tools or analytical techniques about the found security flaws in the smart contracts.
[[2212.05101] Reconfigurable Intelligent Surfaces: The New Frontier of Next G Security](http://arxiv.org/abs/2212.05101) #securityRIS is one of the significant technological advancements that will mark next-generation wireless. RIS technology also opens up the possibility of new security threats, since the reflection of impinging signals can be used for malicious purposes. This article introduces the basic concept for a RIS-assisted attack that re-uses the legitimate signal towards a malicious objective. Specific attacks are identified from this base scenario, and the RIS-assisted signal cancellation attack is selected for evaluation as an attack that inherently exploits RIS capabilities. The key takeaway from the evaluation is that an effective attack requires accurate channel information, a RIS deployed in a favorable location (from the point of view of the attacker), and it disproportionately affects legitimate links that already suffer from reduced path loss. These observations motivate specific security solutions and recommendations for future work.
[[2212.05111] SoK: MEV Countermeasures: Theory and Practice](http://arxiv.org/abs/2212.05111) #securityBlockchains offer strong security guarantees, but they cannot protect the ordering of transactions. Powerful players, such as miners, sequencers, and sophisticated bots, can reap significant profits by selectively including, excluding, or re-ordering user transactions. Such profits are called Miner/Maximal Extractable Value or MEV. MEV bears profound implications for blockchain security and decentralization. While numerous countermeasures have been proposed, there is no agreement on the best solution. Moreover, solutions developed in academic literature differ quite drastically from what is widely adopted by practitioners. For these reasons, this paper systematizes the knowledge of the theory and practice of MEV countermeasures. The contribution is twofold. First, we present a comprehensive taxonomy of 28 proposed MEV countermeasures, covering four different technical directions. Secondly, we empirically studied the most popular MEV- auction-based solution with rich blockchain and mempool data. In addition to gaining insights into MEV auction platforms' real-world operations, our study shed light on the prevalent censorship by MEV auction platforms as a result of the recent OFAC sanction, and its implication on blockchain properties.
[[2212.05310] A systematic literature review on cyber threat hunting](http://arxiv.org/abs/2212.05310) #securitySince the term "Cyber threat hunting" was introduced in 2016, there have been a rising trend of proactive defensive measure to create more cyber security. This research will look into peer reviewed literature on the subject of cyber threat hunting. Our study shows an increase in the field with methods of machine learning.\ Keywords: Cyber threat, Cyber security, threat hunting , security system, data driven, Intel, analytic driven, TTPs
[[2212.05347] A systematic literature review on insider threats](http://arxiv.org/abs/2212.05347) #securityInsider threats is the most concerned cybersecurity problem which is poorly addressed by widely used security solutions. Despite the fact that there have been several scientific publications in this area, but from our innovative study classification and structural taxonomy proposals, we argue to provide the more information about insider threats and defense measures used to counter them. While adopting the current grounded theory method for a thorough literature evaluation, our categorization's goal is to organize knowledge in insider threat research. Along with an analysis of major recent studies on detecting insider threats, the major goal of the study is to develop a classification of current types of insiders, levels of access, motivations behind it, insider profiling, security properties, and methods they use to attack. This includes use of machine learning algorithm, behavior analysis, methods of detection and evaluation. Moreover, actual incidents related to insider attacks have also been analyzed.
[[2212.05438] Understanding Concurrency Vulnerabilities in Linux Kernel](http://arxiv.org/abs/2212.05438) #securityWhile there is a large body of work on analyzing concurrency related software bugs and developing techniques for detecting and patching them, little attention has been given to concurrency related security vulnerabilities. The two are different in that not all bugs are vulnerabilities: for a bug to be exploitable, there needs be a way for attackers to trigger its execution and cause damage, e.g., by revealing sensitive data or running malicious code. To fill the gap, we conduct the first empirical study of concurrency vulnerabilities reported in the Linux operating system in the past ten years. We focus on analyzing the confirmed vulnerabilities archived in the Common Vulnerabilities and Exposures (CVE) database, which are then categorized into different groups based on bug types, exploit patterns, and patch strategies adopted by developers. We use code snippets to illustrate individual vulnerability types and patch strategies. We also use statistics to illustrate the entire landscape, including the percentage of each vulnerability type. We hope to shed some light on the problem, e.g., concurrency vulnerabilities continue to pose a serious threat to system security, and it is difficult even for kernel developers to analyze and patch them. Therefore, more efforts are needed to develop tools and techniques for analyzing and patching these vulnerabilities.
[[2212.05614] Generic Tagging for RISC-V Binaries](http://arxiv.org/abs/2212.05614) #securityWith the widespread popularity of RISC-V -- an open-source ISA -- custom hardware security solutions targeting specific defense needs are gaining popularity. These solutions often require specialized compilers that can insert metadata (called tags) into the generated binaries, and/or extend the RISC-V ISA with new instructions. Developing such compilers can be a tedious and time-consuming process. In this paper, we present COGENT, a generic instruction tag generator for RISC-V architecture. COGENT is capable of associating a tag of configurable and varying widths (1 to 20 bits) to each instruction. It is also capable of emitting labels that are central to the implementation of control-flow integrity (CFI) solutions. COGENT encodes all tags and labels as nop instructions thereby providing full backward compatibility.
We evaluate COGENT on a subset of programs from the SPEC CPU2017 benchmark suite and report the binary size increase to be 29.3% and 18.27% for the lowest and highest tag coverage levels respectively. Additionally, we executed tagged programs on COTS RISC-V unmodified hardware and found the execution time overhead (with respect to backward compatibility) to be 13.4% and 5.72% for the lowest and highest coverage levels respectively. Finally, using a case study, we present possible use case scenarios where COGENT can be applied.
[[2212.05253] Graph Analysis in Decentralized Online Social Networks with Fine-Grained Privacy Protection](http://arxiv.org/abs/2212.05253) #privacyGraph analysts cannot directly obtain the global structure in decentralized social networks, and analyzing such a network requires collecting local views of the social graph from individual users. Since the edges between users may reveal sensitive social interactions in the local view, applying differential privacy in the data collection process is often desirable, which provides strong and rigorous privacy guarantees. In practical decentralized social graphs, different edges have different privacy requirements due to the distinct sensitivity levels. However, the existing differentially private analysis of social graphs provide the same protection for all edges. To address this issue, this work proposes a fine-grained privacy notion as well as novel algorithms for private graph analysis. We first design a fine-grained relationship differential privacy (FGR-DP) notion for social graph analysis, which enforces different protections for the edges with distinct privacy requirements. Then, we design algorithms for triangle counting and k-stars counting, respectively, which can accurately estimate subgraph counts given fine-grained protection for social edges. We also analyze upper bounds on the estimation error, including k-stars and triangle counts, and show their superior performance compared with the state-of-the-arts. Finally, we perform extensive experiments on two real social graph datasets and demonstrate that the proposed mechanisms satisfying FGR-DP have better utility than the state-of-the-art mechanisms due to the finer-grained protection.
[[2212.05428] ezDPS: An Efficient and Zero-Knowledge Machine Learning Inference Pipeline](http://arxiv.org/abs/2212.05428) #privacyMachine Learning as a service (MLaaS) permits resource-limited clients to access powerful data analytics services ubiquitously. Despite its merits, MLaaS poses significant concerns regarding the integrity of delegated computation and the privacy of the server's model parameters. To address this issue, Zhang et al. (CCS'20) initiated the study of zero-knowledge Machine Learning (zkML). Few zkML schemes have been proposed afterward; however, they focus on sole ML classification algorithms that may not offer satisfactory accuracy or require large-scale training data and model parameters, which may not be desirable for some applications. We propose ezDPS, a new efficient and zero-knowledge ML inference scheme. Unlike prior works, ezDPS is a zkML pipeline in which the data is processed in multiple stages for high accuracy. Each stage of ezDPS is harnessed with an established ML algorithm that is shown to be effective in various applications, including Discrete Wavelet Transformation, Principal Components Analysis, and Support Vector Machine. We design new gadgets to prove ML operations effectively. We fully implemented ezDPS and assessed its performance on real datasets. Experimental results showed that ezDPS achieves one-to-three orders of magnitude more efficient than the generic circuit-based approach in all metrics while maintaining more desirable accuracy than single ML classification approaches.
[[2212.05796] Generalizing DP-SGD with Shuffling and Batching Clipping](http://arxiv.org/abs/2212.05796) #privacyClassical differential private DP-SGD implements individual clipping with random subsampling, which forces a mini-batch SGD approach. We provide a general differential private algorithmic framework that goes beyond DP-SGD and allows any possible first order optimizers (e.g., classical SGD and momentum based SGD approaches) in combination with batch clipping, which clips an aggregate of computed gradients rather than summing clipped gradients (as is done in individual clipping). The framework also admits sampling techniques beyond random subsampling such as shuffling.
Our DP analysis follows the $f$-DP approach and introduces a new proof technique which allows us to also analyse group privacy. In particular, for $E$ epochs work and groups of size $g$, we show a $\sqrt{g E}$ DP dependency for batch clipping with shuffling. This is much better than the previously anticipated linear dependency in $g$ and is much better than the previously expected square root dependency on the total number of rounds within $E$ epochs which is generally much more than $\sqrt{E}$.
[[2212.05643] Detecting Code Injections in Noisy Environments Through EM Signal Analysis and SVD Denoising](http://arxiv.org/abs/2212.05643) #protectThe penetration of embedded devices in networks that support critical applications has rendered them a lucrative target for attackers and evildoers. However, traditional protection mechanisms may not be supported due to the memory and computational limitations of these systems. Recently, the analysis of electromagnetic (EM) emanations has gathered the interest of the research community. Thus, analogous protection systems have emerged as a viable solution e.g., for providing external, non-intrusive control-flow attestation for resource-constrained devices. Unfortunately, the majority of current work fails to account for the implications of real-life factors, predominantly the impact of environmental noise. In this work, we introduce a framework that integrates singular value decomposition (SVD) along with outlier detection for discovering malicious modifications of embedded software even under variable conditions of noise. Our proposed framework achieves high detection accuracy i.e., above 93\% AUC score for unknown attacks, even for extreme noise conditions i.e., -10 SNR. To the best of our knowledge, this is the first time this realistic limiting factor, i.e., environmental noise, is successfully addressed in the context of EM-based anomaly detection for embedded devices.
[[2212.05387] General Adversarial Defense Against Black-box Attacks via Pixel Level and Feature Level Distribution Alignments](http://arxiv.org/abs/2212.05387) #defenseDeep Neural Networks (DNNs) are vulnerable to the black-box adversarial attack that is highly transferable. This threat comes from the distribution gap between adversarial and clean samples in feature space of the target DNNs. In this paper, we use Deep Generative Networks (DGNs) with a novel training mechanism to eliminate the distribution gap. The trained DGNs align the distribution of adversarial samples with clean ones for the target DNNs by translating pixel values. Different from previous work, we propose a more effective pixel level training constraint to make this achievable, thus enhancing robustness on adversarial samples. Further, a class-aware feature-level constraint is formulated for integrated distribution alignment. Our approach is general and applicable to multiple tasks, including image classification, semantic segmentation, and object detection. We conduct extensive experiments on different datasets. Our strategy demonstrates its unique effectiveness and generality against black-box attacks.
[[2212.05630] DISCO: Adversarial Defense with Local Implicit Functions](http://arxiv.org/abs/2212.05630) #defenseThe problem of adversarial defenses for image classification, where the goal is to robustify a classifier against adversarial examples, is considered. Inspired by the hypothesis that these examples lie beyond the natural image manifold, a novel aDversarIal defenSe with local impliCit functiOns (DISCO) is proposed to remove adversarial perturbations by localized manifold projections. DISCO consumes an adversarial image and a query pixel location and outputs a clean RGB value at the location. It is implemented with an encoder and a local implicit module, where the former produces per-pixel deep features and the latter uses the features in the neighborhood of query pixel for predicting the clean RGB value. Extensive experiments demonstrate that both DISCO and its cascade version outperform prior defenses, regardless of whether the defense is known to the attacker. DISCO is also shown to be data and parameter efficient and to mount defenses that transfers across datasets, classifiers and attacks.
[[2212.05667] Fighting Malicious Media Data: A Survey on Tampering Detection and Deepfake Detection](http://arxiv.org/abs/2212.05667) #defenseOnline media data, in the forms of images and videos, are becoming mainstream communication channels. However, recent advances in deep learning, particularly deep generative models, open the doors for producing perceptually convincing images and videos at a low cost, which not only poses a serious threat to the trustworthiness of digital information but also has severe societal implications. This motivates a growing interest of research in media tampering detection, i.e., using deep learning techniques to examine whether media data have been maliciously manipulated. Depending on the content of the targeted images, media forgery could be divided into image tampering and Deepfake techniques. The former typically moves or erases the visual elements in ordinary images, while the latter manipulates the expressions and even the identity of human faces. Accordingly, the means of defense include image tampering detection and Deepfake detection, which share a wide variety of properties. In this paper, we provide a comprehensive review of the current media tampering detection approaches, and discuss the challenges and trends in this field for future research.
[[2212.05680] REAP: A Large-Scale Realistic Adversarial Patch Benchmark](http://arxiv.org/abs/2212.05680) #defenseMachine learning models are known to be susceptible to adversarial perturbation. One famous attack is the adversarial patch, a sticker with a particularly crafted pattern that makes the model incorrectly predict the object it is placed on. This attack presents a critical threat to cyber-physical systems that rely on cameras such as autonomous cars. Despite the significance of the problem, conducting research in this setting has been difficult; evaluating attacks and defenses in the real world is exceptionally costly while synthetic data are unrealistic. In this work, we propose the REAP (REalistic Adversarial Patch) benchmark, a digital benchmark that allows the user to evaluate patch attacks on real images, and under real-world conditions. Built on top of the Mapillary Vistas dataset, our benchmark contains over 14,000 traffic signs. Each sign is augmented with a pair of geometric and lighting transformations, which can be used to apply a digitally generated patch realistically onto the sign. Using our benchmark, we perform the first large-scale assessments of adversarial patch attacks under realistic conditions. Our experiments suggest that adversarial patch attacks may present a smaller threat than previously believed and that the success rate of an attack on simpler digital simulations is not predictive of its actual effectiveness in practice. We release our benchmark publicly at https://github.com/wagner-group/reap-benchmark.
[[2212.05827] Carpet-bombing patch: attacking a deep network without usual requirements](http://arxiv.org/abs/2212.05827) #attackAlthough deep networks have shown vulnerability to evasion attacks, such attacks have usually unrealistic requirements. Recent literature discussed the possibility to remove or not some of these requirements. This paper contributes to this literature by introducing a carpet-bombing patch attack which has almost no requirement. Targeting the feature representations, this patch attack does not require knowing the network task. This attack decreases accuracy on Imagenet, mAP on Pascal Voc, and IoU on Cityscapes without being aware that the underlying tasks involved classification, detection or semantic segmentation, respectively. Beyond the potential safety issues raised by this attack, the impact of the carpet-bombing attack highlights some interesting property of deep network layer dynamic.
[[2212.05197] Formal Model-Driven Analysis of Resilience of GossipSub to Attacks from Misbehaving Peers](http://arxiv.org/abs/2212.05197) #attackGossipSub is a new peer-to-peer communication protocol designed to counter attacks from misbehaving peers by carefully controlling what information is disseminated and to whom, via a score function computed by each peer that captures positive and negative behaviors of its neighbors. The score function depends on several parameters (weights, caps, thresholds, etc.) that can be configured by applications using GossipSub. The specification for GossipSub is written in English and its resilience to attacks from misbehaving peers is supported empirically by emulation testing using an implementation in Golang.
In this work we take a foundational approach to understanding the resilience of GossipSub to attacks from misbehaving peers. We build the first formal model of GossipSub, using the ACL2s theorem prover. Our model is officially endorsed by GossipSub developers. It can simulate GossipSub networks of arbitrary size and topology, with arbitrarily configured peers, and can be used to prove and disprove theorems about the protocol. We formalize fundamental security properties stating that the score function is fair, penalizes bad behavior and rewards good behavior. We prove that the score function is always fair, but can be configured in ways that either penalize good behavior or ignore bad behavior. Using our model, we run GossipSub with the specific configurations for two popular real-world applications: the FileCoin and Eth2.0 blockchains. We show that all properties hold for FileCoin. However, given any Eth2.0 network (of any topology and size) with any number of potentially misbehaving peers, we can synthesize attacks where these peers are able to continuously misbehave by never forwarding topic messages, while maintaining positive scores so that they are never pruned from the network by GossipSub.
[[2212.05358] Efficient and Generic Algorithms for Quantitative Attack Tree Analysis](http://arxiv.org/abs/2212.05358) #attackNumerous analysis methods for quantitative attack tree analysis have been proposed. These algorithms compute relevant security metrics, i.e. performance indicators that quantify how good the security of a system is; typical metrics being the most likely attack, the cheapest, or the most damaging one. However, existing methods are only geared towards specific metrics or do not work on general attack trees. This paper classifies attack trees in two dimensions: proper trees vs. directed acyclic graphs (i.e. with shared subtrees); and static vs. dynamic gates. For three out of these four classes, we propose novel algorithms that work over a generic attribute domain, encompassing a large number of concrete security metrics defined on the attack tree semantics; dynamic attack trees with directed acyclic graph structure are left as an open problem. We also analyse the computational complexity of our methods.
[[2212.05380] Mitigating Adversarial Gray-Box Attacks Against Phishing Detectors](http://arxiv.org/abs/2212.05380) #attackAlthough machine learning based algorithms have been extensively used for detecting phishing websites, there has been relatively little work on how adversaries may attack such "phishing detectors" (PDs for short). In this paper, we propose a set of Gray-Box attacks on PDs that an adversary may use which vary depending on the knowledge that he has about the PD. We show that these attacks severely degrade the effectiveness of several existing PDs. We then propose the concept of operation chains that iteratively map an original set of features to a new set of features and develop the "Protective Operation Chain" (POC for short) algorithm. POC leverages the combination of random feature selection and feature mappings in order to increase the attacker's uncertainty about the target PD. Using 3 existing publicly available datasets plus a fourth that we have created and will release upon the publication of this paper, we show that POC is more robust to these attacks than past competing work, while preserving predictive performance when no adversarial attacks are present. Moreover, POC is robust to attacks on 13 different classifiers, not just one. These results are shown to be statistically significant at the p < 0.001 level.
[[2212.05337] Targeted Adversarial Attacks on Deep Reinforcement Learning Policies via Model Checking](http://arxiv.org/abs/2212.05337) #attackDeep Reinforcement Learning (RL) agents are susceptible to adversarial noise in their observations that can mislead their policies and decrease their performance. However, an adversary may be interested not only in decreasing the reward, but also in modifying specific temporal logic properties of the policy. This paper presents a metric that measures the exact impact of adversarial attacks against such properties. We use this metric to craft optimal adversarial attacks. Furthermore, we introduce a model checking method that allows us to verify the robustness of RL policies against adversarial attacks. Our empirical analysis confirms (1) the quality of our metric to craft adversarial attacks against temporal logic properties, and (2) that we are able to concisely assess a system's robustness against attacks.
[[2212.05223] Progressive Multi-view Human Mesh Recovery with Self-Supervision](http://arxiv.org/abs/2212.05223) #robustTo date, little attention has been given to multi-view 3D human mesh estimation, despite real-life applicability (e.g., motion capture, sport analysis) and robustness to single-view ambiguities. Existing solutions typically suffer from poor generalization performance to new settings, largely due to the limited diversity of image-mesh pairs in multi-view training data. To address this shortcoming, people have explored the use of synthetic images. But besides the usual impact of visual gap between rendered and target data, synthetic-data-driven multi-view estimators also suffer from overfitting to the camera viewpoint distribution sampled during training which usually differs from real-world distributions. Tackling both challenges, we propose a novel simulation-based training pipeline for multi-view human mesh recovery, which (a) relies on intermediate 2D representations which are more robust to synthetic-to-real domain gap; (b) leverages learnable calibration and triangulation to adapt to more diversified camera setups; and (c) progressively aggregates multi-view information in a canonical 3D space to remove ambiguities in 2D representations. Through extensive benchmarking, we demonstrate the superiority of the proposed solution especially for unseen in-the-wild scenarios.
[[2212.05262] Position Embedding Needs an Independent Layer Normalization](http://arxiv.org/abs/2212.05262) #robustThe Position Embedding (PE) is critical for Vision Transformers (VTs) due to the permutation-invariance of self-attention operation. By analyzing the input and output of each encoder layer in VTs using reparameterization and visualization, we find that the default PE joining method (simply adding the PE and patch embedding together) operates the same affine transformation to token embedding and PE, which limits the expressiveness of PE and hence constrains the performance of VTs. To overcome this limitation, we propose a simple, effective, and robust method. Specifically, we provide two independent layer normalizations for token embeddings and PE for each layer, and add them together as the input of each layer's Muti-Head Self-Attention module. Since the method allows the model to adaptively adjust the information of PE for different layers, we name it as Layer-adaptive Position Embedding, abbreviated as LaPE. Extensive experiments demonstrate that LaPE can improve various VTs with different types of PE and make VTs robust to PE types. For example, LaPE improves 0.94% accuracy for ViT-Lite on Cifar10, 0.98% for CCT on Cifar100, and 1.72% for DeiT on ImageNet-1K, which is remarkable considering the negligible extra parameters, memory and computational cost brought by LaPE. The code is publicly available at https://github.com/Ingrid725/LaPE.
[[2212.05332] An approach to robust ICP initialization](http://arxiv.org/abs/2212.05332) #robustIn this note, we propose an approach for initializing the Iterative Closest Point (ICP) algorithm that allows us to apply ICP to unlabelled point clouds that are related by rigid transformations. We also give bounds on the robustness of our approach to noise. Numerical experiments confirm our theoretical findings.
[[2212.05463] Vision Transformer with Attentive Pooling for Robust Facial Expression Recognition](http://arxiv.org/abs/2212.05463) #robustFacial Expression Recognition (FER) in the wild is an extremely challenging task. Recently, some Vision Transformers (ViT) have been explored for FER, but most of them perform inferiorly compared to Convolutional Neural Networks (CNN). This is mainly because the new proposed modules are difficult to converge well from scratch due to lacking inductive bias and easy to focus on the occlusion and noisy areas. TransFER, a representative transformer-based method for FER, alleviates this with multi-branch attention dropping but brings excessive computations. On the contrary, we present two attentive pooling (AP) modules to pool noisy features directly. The AP modules include Attentive Patch Pooling (APP) and Attentive Token Pooling (ATP). They aim to guide the model to emphasize the most discriminative features while reducing the impacts of less relevant features. The proposed APP is employed to select the most informative patches on CNN features, and ATP discards unimportant tokens in ViT. Being simple to implement and without learnable parameters, the APP and ATP intuitively reduce the computational cost while boosting the performance by ONLY pursuing the most discriminative features. Qualitative results demonstrate the motivations and effectiveness of our attentive poolings. Besides, quantitative results on six in-the-wild datasets outperform other state-of-the-art methods.
[[2212.05598] Recurrent Vision Transformers for Object Detection with Event Cameras](http://arxiv.org/abs/2212.05598) #robustWe present Recurrent Vision Transformers (RVTs), a novel backbone for object detection with event cameras. Event cameras provide visual information with sub-millisecond latency at a high-dynamic range and with strong robustness against motion blur. These unique properties offer great potential for low-latency object detection and tracking in time-critical scenarios. Prior work in event-based vision has achieved outstanding detection performance but at the cost of substantial inference time, typically beyond 40 milliseconds. By revisiting the high-level design of recurrent vision backbones, we reduce inference time by a factor of 5 while retaining similar performance. To achieve this, we explore a multi-stage design that utilizes three key concepts in each stage: First, a convolutional prior that can be regarded as a conditional positional embedding. Second, local- and dilated global self-attention for spatial feature interaction. Third, recurrent temporal feature aggregation to minimize latency while retaining temporal information. RVTs can be trained from scratch to reach state-of-the-art performance on event-based object detection - achieving an mAP of 47.5% on the Gen1 automotive dataset. At the same time, RVTs offer fast inference (13 ms on a T4 GPU) and favorable parameter efficiency (5 times fewer than prior art). Our study brings new insights into effective design choices that could be fruitful for research beyond event-based vision.
[[2212.05679] Evolutionary Multitasking with Solution Space Cutting for Point Cloud Registration](http://arxiv.org/abs/2212.05679) #robustPoint cloud registration (PCR) is a popular research topic in computer vision. Recently, the registration method in an evolutionary way has received continuous attention because of its robustness to the initial pose and flexibility in objective function design. However, most evolving registration methods cannot tackle the local optimum well and they have rarely investigated the success ratio, which implies the probability of not falling into local optima and is closely related to the practicality of the algorithm. Evolutionary multi-task optimization (EMTO) is a widely used paradigm, which can boost exploration capability through knowledge transfer among related tasks. Inspired by this concept, this study proposes a novel evolving registration algorithm via EMTO, where the multi-task configuration is based on the idea of solution space cutting. Concretely, one task searching in cut space assists another task with complex function landscape in escaping from local optima and enhancing successful registration ratio. To reduce unnecessary computational cost, a sparse-to-dense strategy is proposed. In addition, a novel fitness function robust to various overlap rates as well as a problem-specific metric of computational cost is introduced. Compared with 7 evolving registration approaches and 4 traditional registration approaches on the object-scale and scene-scale registration datasets, experimental results demonstrate that the proposed method has superior performances in terms of precision and tackling local optima.
[[2212.05691] CircleNet: Reciprocating Feature Adaptation for Robust Pedestrian Detection](http://arxiv.org/abs/2212.05691) #robustPedestrian detection in the wild remains a challenging problem especially when the scene contains significant occlusion and/or low resolution of the pedestrians to be detected. Existing methods are unable to adapt to these difficult cases while maintaining acceptable performance. In this paper we propose a novel feature learning model, referred to as CircleNet, to achieve feature adaptation by mimicking the process humans looking at low resolution and occluded objects: focusing on it again, at a finer scale, if the object can not be identified clearly for the first time. CircleNet is implemented as a set of feature pyramids and uses weight sharing path augmentation for better feature fusion. It targets at reciprocating feature adaptation and iterative object detection using multiple top-down and bottom-up pathways. To take full advantage of the feature adaptation capability in CircleNet, we design an instance decomposition training strategy to focus on detecting pedestrian instances of various resolutions and different occlusion levels in each cycle. Specifically, CircleNet implements feature ensemble with the idea of hard negative boosting in an end-to-end manner. Experiments on two pedestrian detection datasets, Caltech and CityPersons, show that CircleNet improves the performance of occluded and low-resolution pedestrians with significant margins while maintaining good performance on normal instances.
[[2212.05855] BeautyREC: Robust, Efficient, and Content-preserving Makeup Transfer](http://arxiv.org/abs/2212.05855) #robustIn this work, we propose a Robust, Efficient, and Component-specific makeup transfer method (abbreviated as BeautyREC). A unique departure from prior methods that leverage global attention, simply concatenate features, or implicitly manipulate features in latent space, we propose a component-specific correspondence to directly transfer the makeup style of a reference image to the corresponding components (e.g., skin, lips, eyes) of a source image, making elaborate and accurate local makeup transfer. As an auxiliary, the long-range visual dependencies of Transformer are introduced for effective global makeup transfer. Instead of the commonly used cycle structure that is complex and unstable, we employ a content consistency loss coupled with a content encoder to implement efficient single-path makeup transfer. The key insights of this study are modeling component-specific correspondence for local makeup transfer, capturing long-range dependencies for global makeup transfer, and enabling efficient makeup transfer via a single-path structure. We also contribute BeautyFace, a makeup transfer dataset to supplement existing datasets. This dataset contains 3,000 faces, covering more diverse makeup styles, face poses, and races. Each face has annotated parsing map. Extensive experiments demonstrate the effectiveness of our method against state-of-the-art methods. Besides, our method is appealing as it is with only 1M parameters, outperforming the state-of-the-art methods (BeautyGAN: 8.43M, PSGAN: 12.62M, SCGAN: 15.30M, CPM: 9.24M, SSAT: 10.48M).
[[2212.05895] Diff-Font: Diffusion Model for Robust One-Shot Font Generation](http://arxiv.org/abs/2212.05895) #robustFont generation is a difficult and time-consuming task, especially in those languages using ideograms that have complicated structures with a large number of characters, such as Chinese. To solve this problem, few-shot font generation and even one-shot font generation have attracted a lot of attention. However, most existing font generation methods may still suffer from (i) large cross-font gap challenge; (ii) subtle cross-font variation problem; and (iii) incorrect generation of complicated characters. In this paper, we propose a novel one-shot font generation method based on a diffusion model, named Diff-Font, which can be stably trained on large datasets. The proposed model aims to generate the entire font library by giving only one sample as the reference. Specifically, a large stroke-wise dataset is constructed, and a stroke-wise diffusion model is proposed to preserve the structure and the completion of each generated character. To our best knowledge, the proposed Diff-Font is the first work that developed diffusion models to handle the font generation task. The well-trained Diff-Font is not only robust to font gap and font variation, but also achieved promising performance on difficult character generation. Compared to previous font generation methods, our model reaches state-of-the-art performance both qualitatively and quantitatively.
[[2212.05124] Multi-view Graph Convolutional Networks with Differentiable Node Selection](http://arxiv.org/abs/2212.05124) #robustMulti-view data containing complementary and consensus information can facilitate representation learning by exploiting the intact integration of multi-view features. Because most objects in real world often have underlying connections, organizing multi-view data as heterogeneous graphs is beneficial to extracting latent information among different objects. Due to the powerful capability to gather information of neighborhood nodes, in this paper, we apply Graph Convolutional Network (GCN) to cope with heterogeneous-graph data originating from multi-view data, which is still under-explored in the field of GCN. In order to improve the quality of network topology and alleviate the interference of noises yielded by graph fusion, some methods undertake sorting operations before the graph convolution procedure. These GCN-based methods generally sort and select the most confident neighborhood nodes for each vertex, such as picking the top-k nodes according to pre-defined confidence values. Nonetheless, this is problematic due to the non-differentiable sorting operators and inflexible graph embedding learning, which may result in blocked gradient computations and undesired performance. To cope with these issues, we propose a joint framework dubbed Multi-view Graph Convolutional Network with Differentiable Node Selection (MGCN-DNS), which is constituted of an adaptive graph fusion layer, a graph learning module and a differentiable node selection schema. MGCN-DNS accepts multi-channel graph-structural data as inputs and aims to learn more robust graph fusion through a differentiable neural network. The effectiveness of the proposed method is verified by rigorous comparisons with considerable state-of-the-art approaches in terms of multi-view semi-supervised classification tasks.
[[2212.05331] Effects of Spectral Normalization in Multi-agent Reinforcement Learning](http://arxiv.org/abs/2212.05331) #robustA reliable critic is central to on-policy actor-critic learning. But it becomes challenging to learn a reliable critic in a multi-agent sparse reward scenario due to two factors: 1) The joint action space grows exponentially with the number of agents 2) This, combined with the reward sparseness and environment noise, leads to large sample requirements for accurate learning. We show that regularising the critic with spectral normalization (SN) enables it to learn more robustly, even in multi-agent on-policy sparse reward scenarios. Our experiments show that the regularised critic is quickly able to learn from the sparse rewarding experience in the complex SMAC and RWARE domains. These findings highlight the importance of regularisation in the critic for stable learning.
[[2212.05430] Corruption-tolerant Algorithms for Generalized Linear Models](http://arxiv.org/abs/2212.05430) #robustThis paper presents SVAM (Sequential Variance-Altered MLE), a unified framework for learning generalized linear models under adversarial label corruption in training data. SVAM extends to tasks such as least squares regression, logistic regression, and gamma regression, whereas many existing works on learning with label corruptions focus only on least squares regression. SVAM is based on a novel variance reduction technique that may be of independent interest and works by iteratively solving weighted MLEs over variance-altered versions of the GLM objective. SVAM offers provable model recovery guarantees superior to the state-of-the-art for robust regression even when a constant fraction of training labels are adversarially corrupted. SVAM also empirically outperforms several existing problem-specific techniques for robust regression and classification. Code for SVAM is available at https://github.com/purushottamkar/svam/
[[2212.05502] Estimator: An Effective and Scalable Framework for Transportation Mode Classification over Trajectories](http://arxiv.org/abs/2212.05502) #robustTransportation mode classification, the process of predicting the class labels of moving objects transportation modes, has been widely applied to a variety of real world applications, such as traffic management, urban computing, and behavior study. However, existing studies of transportation mode classification typically extract the explicit features of trajectory data but fail to capture the implicit features that affect the classification performance. In addition, most of the existing studies also prefer to apply RNN-based models to embed trajectories, which is only suitable for classifying small-scale data. To tackle the above challenges, we propose an effective and scalable framework for transportation mode classification over GPS trajectories, abbreviated Estimator. Estimator is established on a developed CNN-TCN architecture, which is capable of leveraging the spatial and temporal hidden features of trajectories to achieve high effectiveness and efficiency. Estimator partitions the entire traffic space into disjointed spatial regions according to traffic conditions, which enhances the scalability significantly and thus enables parallel transportation classification. Extensive experiments using eight public real-life datasets offer evidence that Estimator i) achieves superior model effectiveness (i.e., 99% Accuracy and 0.98 F1-score), which outperforms state-of-the-arts substantially; ii) exhibits prominent model efficiency, and obtains 7-40x speedups up over state-of-the-arts learning-based methods; and iii) shows high model scalability and robustness that enables large-scale classification analytics.
[[2212.05662] Optimal Planning of Hybrid Energy Storage Systems using Curtailed Renewable Energy through Deep Reinforcement Learning](http://arxiv.org/abs/2212.05662) #robustEnergy management systems (EMS) are becoming increasingly important in order to utilize the continuously growing curtailed renewable energy. Promising energy storage systems (ESS), such as batteries and green hydrogen should be employed to maximize the efficiency of energy stakeholders. However, optimal decision-making, i.e., planning the leveraging between different strategies, is confronted with the complexity and uncertainties of large-scale problems. Here, we propose a sophisticated deep reinforcement learning (DRL) methodology with a policy-based algorithm to realize the real-time optimal ESS planning under the curtailed renewable energy uncertainty. A quantitative performance comparison proved that the DRL agent outperforms the scenario-based stochastic optimization (SO) algorithm, even with a wide action and observation space. Owing to the uncertainty rejection capability of the DRL, we could confirm a robust performance, under a large uncertainty of the curtailed renewable energy, with a maximizing net profit and stable system. Action-mapping was performed for visually assessing the action taken by the DRL agent according to the state. The corresponding results confirmed that the DRL agent learns the way like what a human expert would do, suggesting reliable application of the proposed methodology.
[[2212.05716] On Generalization and Regularization via Wasserstein Distributionally Robust Optimization](http://arxiv.org/abs/2212.05716) #robustWasserstein distributionally robust optimization (DRO) has found success in operations research and machine learning applications as a powerful means to obtain solutions with favourable out-of-sample performances. Two compelling explanations for the success are the generalization bounds derived from Wasserstein DRO and the equivalency between Wasserstein DRO and the regularization scheme commonly applied in machine learning. Existing results on generalization bounds and the equivalency to regularization are largely limited to the setting where the Wasserstein ball is of a certain type and the decision criterion takes certain forms of an expected function. In this paper, we show that by focusing on Wasserstein DRO problems with affine decision rules, it is possible to obtain generalization bounds and the equivalency to regularization in a significantly broader setting where the Wasserstein ball can be of a general type and the decision criterion can be a general measure of risk, i.e., nonlinear in distributions. This allows for accommodating many important classification, regression, and risk minimization applications that have not been addressed to date using Wasserstein DRO. Our results are strong in that the generalization bounds do not suffer from the curse of dimensionality and the equivalency to regularization is exact. As a byproduct, our regularization results broaden considerably the class of Wasserstein DRO models that can be solved efficiently via regularization formulations.
[[2212.05727] Evaluating Model-free Reinforcement Learning toward Safety-critical Tasks](http://arxiv.org/abs/2212.05727) #robustSafety comes first in many real-world applications involving autonomous agents. Despite a large number of reinforcement learning (RL) methods focusing on safety-critical tasks, there is still a lack of high-quality evaluation of those algorithms that adheres to safety constraints at each decision step under complex and unknown dynamics. In this paper, we revisit prior work in this scope from the perspective of state-wise safe RL and categorize them as projection-based, recovery-based, and optimization-based approaches, respectively. Furthermore, we propose Unrolling Safety Layer (USL), a joint method that combines safety optimization and safety projection. This novel technique explicitly enforces hard constraints via the deep unrolling architecture and enjoys structural advantages in navigating the trade-off between reward improvement and constraint satisfaction. To facilitate further research in this area, we reproduce related algorithms in a unified pipeline and incorporate them into SafeRL-Kit, a toolkit that provides off-the-shelf interfaces and evaluation utilities for safety-critical tasks. We then perform a comparative study of the involved algorithms on six benchmarks ranging from robotic control to autonomous driving. The empirical results provide an insight into their applicability and robustness in learning zero-cost-return policies without task-dependent handcrafting. The project page is available at https://sites.google.com/view/saferlkit.
[[2212.05781] Robust Recurrent Neural Network to Identify Ship Motion in Open Water with Performance Guarantees -- Technical Report](http://arxiv.org/abs/2212.05781) #robustRecurrent neural networks are capable of learning the dynamics of an unknown nonlinear system purely from input-output measurements. However, the resulting models do not provide any stability guarantees on the input-output mapping. In this work, we represent a recurrent neural network as a linear time-invariant system with nonlinear disturbances. By introducing constraints on the parameters, we can guarantee finite gain stability and incremental finite gain stability. We apply this identification method to learn the motion of a four-degrees-of-freedom ship that is moving in open water and compare it against other purely learning-based approaches with unconstrained parameters. Our analysis shows that the constrained recurrent neural network has a lower prediction accuracy on the test set, but it achieves comparable results on an out-of-distribution set and respects stability conditions.
[[2212.05884] Finger-NestNet: Interpretable Fingerphoto Verification on Smartphone using Deep Nested Residual Network](http://arxiv.org/abs/2212.05884) #biometricFingerphoto images captured using a smartphone are successfully used to verify the individuals that have enabled several applications. This work presents a novel algorithm for fingerphoto verification using a nested residual block: Finger-NestNet. The proposed Finger-NestNet architecture is designed with three consecutive convolution blocks followed by a series of nested residual blocks to achieve reliable fingerphoto verification. This paper also presents the interpretability of the proposed method using four different visualization techniques that can shed light on the critical regions in the fingerphoto biometrics that can contribute to the reliable verification performance of the proposed method. Extensive experiments are performed on the fingerphoto dataset comprised of 196 unique fingers collected from 52 unique data subjects using an iPhone6S. Experimental results indicate the improved verification of the proposed method compared to six different existing methods with EER = 1.15%.
[[2212.05752] Scale-Semantic Joint Decoupling Network for Image-text Retrieval in Remote Sensing](http://arxiv.org/abs/2212.05752) #extractionImage-text retrieval in remote sensing aims to provide flexible information
for data analysis and application. In recent years, state-of-the-art methods
are dedicated to scale decoupling'' andsemantic decoupling'' strategies to
further enhance the capability of representation. However, these previous
approaches focus on either the disentangling scale or semantics but ignore
merging these two ideas in a union model, which extremely limits the
performance of cross-modal retrieval models. To address these issues, we
propose a novel Scale-Semantic Joint Decoupling Network (SSJDN) for remote
sensing image-text retrieval. Specifically, we design the Bidirectional Scale
Decoupling (BSD) module, which exploits Salience Feature Extraction (SFE) and
Salience-Guided Suppression (SGS) units to adaptively extract potential
features and suppress cumbersome features at other scales in a bidirectional
pattern to yield different scale clues. Besides, we design the Label-supervised
Semantic Decoupling (LSD) module by leveraging the category semantic labels as
prior knowledge to supervise images and texts probing significant
semantic-related information. Finally, we design a Semantic-guided Triple Loss
(STL), which adaptively generates a constant to adjust the loss function to
improve the probability of matching the same semantic image and text and
shorten the convergence time of the retrieval model. Our proposed SSJDN
outperforms state-of-the-art approaches in numerical experiments conducted on
four benchmark remote sensing datasets.
[[2212.05238] Structured information extraction from complex scientific text with fine-tuned large language models](http://arxiv.org/abs/2212.05238) #extractionIntelligently extracting and linking complex scientific information from unstructured text is a challenging endeavor particularly for those inexperienced with natural language processing. Here, we present a simple sequence-to-sequence approach to joint named entity recognition and relation extraction for complex hierarchical information in scientific text. The approach leverages a pre-trained large language model (LLM), GPT-3, that is fine-tuned on approximately 500 pairs of prompts (inputs) and completions (outputs). Information is extracted either from single sentences or across sentences in abstracts/passages, and the output can be returned as simple English sentences or a more structured format, such as a list of JSON objects. We demonstrate that LLMs trained in this way are capable of accurately extracting useful records of complex scientific knowledge for three representative tasks in materials chemistry: linking dopants with their host materials, cataloging metal-organic frameworks, and general chemistry/phase/morphology/application information extraction. This approach represents a simple, accessible, and highly-flexible route to obtaining large databases of structured knowledge extracted from unstructured text. An online demo is available at this http URL
[[2212.05429] MORTY: Structured Summarization for Targeted Information Extraction from Scholarly Articles](http://arxiv.org/abs/2212.05429) #extractionInformation extraction from scholarly articles is a challenging task due to the sizable document length and implicit information hidden in text, figures, and citations. Scholarly information extraction has various applications in exploration, archival, and curation services for digital libraries and knowledge management systems. We present MORTY, an information extraction technique that creates structured summaries of text from scholarly articles. Our approach condenses the article's full-text to property-value pairs as a segmented text snippet called structured summary. We also present a sizable scholarly dataset combining structured summaries retrieved from a scholarly knowledge graph and corresponding publicly available scientific articles, which we openly publish as a resource for the research community. Our results show that structured summarization is a suitable approach for targeted information extraction that complements other commonly used methods such as question answering and named entity recognition.
[[2212.05696] Ensembling Transformers for Cross-domain Automatic Term Extraction](http://arxiv.org/abs/2212.05696) #extractionAutomatic term extraction plays an essential role in domain language understanding and several natural language processing downstream tasks. In this paper, we propose a comparative study on the predictive power of Transformers-based pretrained language models toward term extraction in a multi-language cross-domain setting. Besides evaluating the ability of monolingual models to extract single- and multi-word terms, we also experiment with ensembles of mono- and multilingual models by conducting the intersection or union on the term output sets of different language models. Our experiments have been conducted on the ACTER corpus covering four specialized domains (Corruption, Wind energy, Equitation, and Heart failure) and three languages (English, French, and Dutch), and on the RSDO5 Slovenian corpus covering four additional domains (Biomechanics, Chemistry, Veterinary, and Linguistics). The results show that the strategy of employing monolingual models outperforms the state-of-the-art approaches from the related work leveraging multilingual models, regarding all the languages except Dutch and French if the term extraction task excludes the extraction of named entity terms. Furthermore, by combining the outputs of the two best performing models, we achieve significant improvements.
[[2212.05178] State-Regularized Recurrent Neural Networks to Extract Automata and Explain Predictions](http://arxiv.org/abs/2212.05178) #extractionRecurrent neural networks are a widely used class of neural architectures. They have, however, two shortcomings. First, they are often treated as black-box models and as such it is difficult to understand what exactly they learn as well as how they arrive at a particular prediction. Second, they tend to work poorly on sequences requiring long-term memorization, despite having this capacity in principle. We aim to address both shortcomings with a class of recurrent networks that use a stochastic state transition mechanism between cell applications. This mechanism, which we term state-regularization, makes RNNs transition between a finite set of learnable states. We evaluate state-regularized RNNs on (1) regular languages for the purpose of automata extraction; (2) non-regular languages such as balanced parentheses and palindromes where external memory is required; and (3) real-word sequence learning tasks for sentiment analysis, visual object recognition and text categorisation. We show that state-regularization (a) simplifies the extraction of finite state automata that display an RNN's state transition dynamic; (b) forces RNNs to operate more like automata with external memory and less like finite state machines, which potentiality leads to a more structural memory; (c) leads to better interpretability and explainability of RNNs by leveraging the probabilistic finite state transition mechanism over time steps.
[[2212.05289] A Hybrid Brain-Computer Interface Using Motor Imagery and SSVEP Based on Convolutional Neural Network](http://arxiv.org/abs/2212.05289) #extractionThe key to electroencephalography (EEG)-based brain-computer interface (BCI) lies in neural decoding, and its accuracy can be improved by using hybrid BCI paradigms, that is, fusing multiple paradigms. However, hybrid BCIs usually require separate processing processes for EEG signals in each paradigm, which greatly reduces the efficiency of EEG feature extraction and the generalizability of the model. Here, we propose a two-stream convolutional neural network (TSCNN) based hybrid brain-computer interface. It combines steady-state visual evoked potential (SSVEP) and motor imagery (MI) paradigms. TSCNN automatically learns to extract EEG features in the two paradigms in the training process, and improves the decoding accuracy by 25.4% compared with the MI mode, and 2.6% compared with SSVEP mode in the test data. Moreover, the versatility of TSCNN is verified as it provides considerable performance in both single-mode (70.2% for MI, 93.0% for SSVEP) and hybrid-mode scenarios (95.6% for MI-SSVEP hybrid). Our work will facilitate the real-world applications of EEG-based BCI systems.
[[2212.05789] Collaborating Heterogeneous Natural Language Processing Tasks via Federated Learning](http://arxiv.org/abs/2212.05789) #federateThe increasing privacy concerns on personal private text data promote the development of federated learning (FL) in recent years. However, the existing studies on applying FL in NLP are not suitable to coordinate participants with heterogeneous or private learning objectives. In this study, we further broaden the application scope of FL in NLP by proposing an Assign-Then-Contrast (denoted as ATC) framework, which enables clients with heterogeneous NLP tasks to construct an FL course and learn useful knowledge from each other. Specifically, the clients are suggested to first perform local training with the unified tasks assigned by the server rather than using their own learning objectives, which is called the Assign training stage. After that, in the Contrast training stage, clients train with different local learning objectives and exchange knowledge with other clients who contribute consistent and useful model updates. We conduct extensive experiments on six widely-used datasets covering both Natural Language Understanding (NLU) and Natural Language Generation (NLG) tasks, and the proposed ATC framework achieves significant improvements compared with various baseline methods. The source code is available at \url{https://github.com/alibaba/FederatedScope/tree/master/federatedscope/nlp/hetero_tasks}.
[[2212.05492] Client Selection for Federated Bayesian Learning](http://arxiv.org/abs/2212.05492) #federateDistributed Stein Variational Gradient Descent (DSVGD) is a non-parametric distributed learning framework for federated Bayesian learning, where multiple clients jointly train a machine learning model by communicating a number of non-random and interacting particles with the server. Since communication resources are limited, selecting the clients with most informative local learning updates can improve the model convergence and communication efficiency. In this paper, we propose two selection schemes for DSVGD based on Kernelized Stein Discrepancy (KSD) and Hilbert Inner Product (HIP). We derive the upper bound on the decrease of the global free energy per iteration for both schemes, which is then minimized to speed up the model convergence. We evaluate and compare our schemes with conventional schemes in terms of model accuracy, convergence speed, and stability using various learning tasks and datasets.
[[2212.05602] ResFed: Communication Efficient Federated Learning by Transmitting Deep Compressed Residuals](http://arxiv.org/abs/2212.05602) #federateFederated learning enables cooperative training among massively distributed clients by sharing their learned local model parameters. However, with increasing model size, deploying federated learning requires a large communication bandwidth, which limits its deployment in wireless networks. To address this bottleneck, we introduce a residual-based federated learning framework (ResFed), where residuals rather than model parameters are transmitted in communication networks for training. In particular, we integrate two pairs of shared predictors for the model prediction in both server-to-client and client-to-server communication. By employing a common prediction rule, both locally and globally updated models are always fully recoverable in clients and the server. We highlight that the residuals only indicate the quasi-update of a model in a single inter-round, and hence contain more dense information and have a lower entropy than the model, comparing to model weights and gradients. Based on this property, we further conduct lossy compression of the residuals by sparsification and quantization and encode them for efficient communication. The experimental evaluation shows that our ResFed needs remarkably less communication costs and achieves better accuracy by leveraging less sensitive residuals, compared to standard federated learning. For instance, to train a 4.08 MB CNN model on CIFAR-10 with 10 clients under non-independent and identically distributed (Non-IID) setting, our approach achieves a compression ratio over 700X in each communication round with minimum impact on the accuracy. To reach an accuracy of 70%, it saves around 99% of the total communication volume from 587.61 Mb to 6.79 Mb in up-streaming and to 4.61 Mb in down-streaming on average for all clients.
[[2212.05765] Information-Theoretic Text Hallucination Reduction for Video-grounded Dialogue](http://arxiv.org/abs/2212.05765) #interpretabilityVideo-grounded Dialogue (VGD) aims to decode an answer sentence to a question regarding a given video and dialogue context. Despite the recent success of multi-modal reasoning to generate answer sentences, existing dialogue systems still suffer from a text hallucination problem, which denotes indiscriminate text-copying from input texts without an understanding of the question. This is due to learning spurious correlations from the fact that answer sentences in the dataset usually include the words of input texts, thus the VGD system excessively relies on copying words from input texts by hoping those words to overlap with ground-truth texts. Hence, we design Text Hallucination Mitigating (THAM) framework, which incorporates Text Hallucination Regularization (THR) loss derived from the proposed information-theoretic text hallucination measurement approach. Applying THAM with current dialogue systems validates the effectiveness on VGD benchmarks (i.e., AVSD@DSTC7 and AVSD@DSTC8) and shows enhanced interpretability.
[[2212.05772] Multi-Dimensional Self Attention based Approach for Remaining Useful Life Estimation](http://arxiv.org/abs/2212.05772) #interpretabilityRemaining Useful Life (RUL) estimation plays a critical role in Prognostics and Health Management (PHM). Traditional machine health maintenance systems are often costly, requiring sufficient prior expertise, and are difficult to fit into highly complex and changing industrial scenarios. With the widespread deployment of sensors on industrial equipment, building the Industrial Internet of Things (IIoT) to interconnect these devices has become an inexorable trend in the development of the digital factory. Using the device's real-time operational data collected by IIoT to get the estimated RUL through the RUL prediction algorithm, the PHM system can develop proactive maintenance measures for the device, thus, reducing maintenance costs and decreasing failure times during operation. This paper carries out research into the remaining useful life prediction model for multi-sensor devices in the IIoT scenario. We investigated the mainstream RUL prediction models and summarized the basic steps of RUL prediction modeling in this scenario. On this basis, a data-driven approach for RUL estimation is proposed in this paper. It employs a Multi-Head Attention Mechanism to fuse the multi-dimensional time-series data output from multiple sensors, in which the attention on features is used to capture the interactions between features and attention on sequences is used to learn the weights of time steps. Then, the Long Short-Term Memory Network is applied to learn the features of time series. We evaluate the proposed model on two benchmark datasets (C-MAPSS and PHM08), and the results demonstrate that it outperforms the state-of-art models. Moreover, through the interpretability of the multi-head attention mechanism, the proposed model can provide a preliminary explanation of engine degradation. Therefore, this approach is promising for predictive maintenance in IIoT scenarios.
[[2212.05199] MAGVIT: Masked Generative Video Transformer](http://arxiv.org/abs/2212.05199) #diffusionWe introduce the MAsked Generative VIdeo Transformer, MAGVIT, to tackle various video synthesis tasks with a single model. We introduce a 3D tokenizer to quantize a video into spatial-temporal visual tokens and propose an embedding method for masked video token modeling to facilitate multi-task learning. We conduct extensive experiments to demonstrate the quality, efficiency, and flexibility of MAGVIT. Our experiments show that (i) MAGVIT performs favorably against state-of-the-art approaches and establishes the best-published FVD on three video generation benchmarks, including the challenging Kinetics-600. (ii) MAGVIT outperforms existing methods in inference time by two orders of magnitude against diffusion models and by 60x against autoregressive models. (iii) A single MAGVIT model supports ten diverse generation tasks and generalizes across videos from different visual domains. The source code and trained models will be released to the public at https://magvit.cs.cmu.edu.
[[2212.05400] How to Backdoor Diffusion Models?](http://arxiv.org/abs/2212.05400) #diffusionDiffusion models are state-of-the-art deep learning empowered generative models that are trained based on the principle of learning forward and reverse diffusion processes via progressive noise-addition and denoising. To gain a better understanding of the limitations and potential risks, this paper presents the first study on the robustness of diffusion models against backdoor attacks. Specifically, we propose BadDiffusion, a novel attack framework that engineers compromised diffusion processes during model training for backdoor implantation. At the inference stage, the backdoored diffusion model will behave just like an untampered generator for regular data inputs, while falsely generating some targeted outcome designed by the bad actor upon receiving the implanted trigger signal. Such a critical risk can be dreadful for downstream tasks and applications built upon the problematic model. Our extensive experiments on various backdoor attack settings show that BadDiffusion can consistently lead to compromised diffusion models with high utility and target specificity. Even worse, BadDiffusion can be made cost-effective by simply finetuning a clean pre-trained diffusion model to implant backdoors. We also explore some possible countermeasures for risk mitigation. Our results call attention to potential risks and possible misuse of diffusion models.
[[2212.05404] DiffAlign : Few-shot learning using diffusion based synthesis and alignment](http://arxiv.org/abs/2212.05404) #diffusionWe address the problem of few-shot classification where the goal is to learn a classifier from a limited set of samples. While data-driven learning is shown to be effective in various applications, learning from less data still remains challenging. To address this challenge, existing approaches consider various data augmentation techniques for increasing the number of training samples. Pseudo-labeling is commonly used in a few-shot setup, where approximate labels are estimated for a large set of unlabeled images. We propose DiffAlign which focuses on generating images from class labels. Specifically, we leverage the recent success of the generative models (e.g., DALL-E and diffusion models) that can generate realistic images from texts. However, naive learning on synthetic images is not adequate due to the domain gap between real and synthetic images. Thus, we employ a maximum mean discrepancy (MMD) loss to align the synthetic images to the real images minimizing the domain gap. We evaluate our method on the standard few-shot classification benchmarks: CIFAR-FS, FC100, miniImageNet, tieredImageNet and a cross-domain few-shot classification benchmark: miniImageNet to CUB. The proposed approach significantly outperforms the stateof-the-art in both 5-shot and 1-shot setups on these benchmarks. Our approach is also shown to be effective in the zero-shot classification setup
[[2212.05707] Human Mobility Modeling During the COVID-19 Pandemic via Deep Graph Diffusion Infomax](http://arxiv.org/abs/2212.05707) #diffusionNon-Pharmaceutical Interventions (NPIs), such as social gathering restrictions, have shown effectiveness to slow the transmission of COVID-19 by reducing the contact of people. To support policy-makers, multiple studies have first modeled human mobility via macro indicators (e.g., average daily travel distance) and then studied the effectiveness of NPIs. In this work, we focus on mobility modeling and, from a micro perspective, aim to predict locations that will be visited by COVID-19 cases. Since NPIs generally cause economic and societal loss, such a micro perspective prediction benefits governments when they design and evaluate them. However, in real-world situations, strict privacy data protection regulations result in severe data sparsity problems (i.e., limited case and location information). To address these challenges, we formulate the micro perspective mobility modeling into computing the relevance score between a diffusion and a location, conditional on a geometric graph. we propose a model named Deep Graph Diffusion Infomax (DGDI), which jointly models variables including a geometric graph, a set of diffusions and a set of locations.To facilitate the research of COVID-19 prediction, we present two benchmarks that contain geometric graphs and location histories of COVID-19 cases. Extensive experiments on the two benchmarks show that DGDI significantly outperforms other competing methods.
[[2212.05782] GT-CausIn: a novel causal-based insight for traffic prediction](http://arxiv.org/abs/2212.05782) #diffusionTraffic forecasting is an important application of spatiotemporal series prediction. Among different methods, graph neural networks have achieved so far the most promising results, learning relations between graph nodes then becomes a crucial task. However, improvement space is very limited when these relations are learned in a node-to-node manner. The challenge stems from (1) obscure temporal dependencies between different stations, (2) difficulties in defining variables beyond the node level, and (3) no ready-made method to validate the learned relations. To confront these challenges, we define legitimate traffic causal variables to discover the causal relation inside the traffic network, which is carefully checked with statistic tools and case analysis. We then present a novel model named Graph Spatial-Temporal Network Based on Causal Insight (GT-CausIn), where prior learned causal information is integrated with graph diffusion layers and temporal convolutional network (TCN) layers. Experiments are carried out on two real-world traffic datasets: PEMS-BAY and METR-LA, which show that GT-CausIn significantly outperforms the state-of-the-art models on mid-term and long-term prediction.