[[2209.06373] SEEK: model extraction attack against hybrid secure inference protocols](http://arxiv.org/abs/2209.06373)Security concerns about a machine learning model used in a prediction-as-a-service include the privacy of the model, the query and the result. Secure inference solutions based on homomorphic encryption (HE) and/or multiparty computation (MPC) have been developed to protect all the sensitive information. One of the most efficient type of solution utilizes HE for linear layers, and MPC for non-linear layers. However, for such hybrid protocols with semi-honest security, an adversary can malleate the intermediate features in the inference process, and extract model information more effectively than methods against inference service in plaintext. In this paper, we propose SEEK, a general extraction method for hybrid secure inference services outputing only class labels. This method can extract each layer of the target model independently, and is not affected by the depth of the model. For ResNet-18, SEEK can extract a parameter with less than 50 queries on average, with average error less than $0.03\%$.
[[2209.06447] Security of Virtual Reality Authentication Methods in Metaverse: An Overview](http://arxiv.org/abs/2209.06447)The metaverse is said to be the future Internet and will consist of several worlds called verses. This concept is being discussed a lot lately, however, the security issues of these virtual worlds are not discussed enough. This study first discusses the privacy and security concerns of the metaverse. Virtual reality headsets are the main devices used to access the Metaverse. The user needs to verify their identity to log in to the metaverse platforms, and the security of this phase becomes vital. This paper aims to compare the security of the main authentication methods that are used in virtual reality environments. Information-based, biometric, and multi-model methods are compared and analyzed in terms of security. These methods aim to verify the user with different data types such as 3D patterns, PIN systems, or biometric data. The pros and cons are discussed. The paper also concludes with what work can be done to improve the safety of these authentication methods and future work.
[[2209.06553] Collaborative SQL-injections detection system with machine learning](http://arxiv.org/abs/2209.06553)Data mining and information extraction from data is a field that has gained relevance in recent years thanks to techniques based on artificial intelligence and use of machine and deep learning. The main aim of the present work is the development of a tool based on a previous behaviour study of security audit tools (oriented to SQL pentesting) with the purpose of creating testing sets capable of performing an accurate detection of a SQL attack. The study is based on the information collected through the generated web server logs in a pentesting laboratory environment. Then, making use of the common extracted patterns from the logs, each attack vector has been classified in risk levels (dangerous attack, normal attack, non-attack, etc.). Finally, a training with the generated data was performed in order to obtain a classifier system that has a variable performance between 97 and 99 percent in positive attack detection. The training data is shared to other servers in order to create a distributed network capable of deciding if a query is an attack or is a real petition and inform to connected clients in order to block the petitions from the attacker's IP.
[[2209.06327] Differentially Private Genomic Data Release For GWAS Reproducibility](http://arxiv.org/abs/2209.06327)With the rapid development of technology in genome-related fields, researchers have proposed various approaches and algorithms in recent years. However, they rarely publish the genomic datasets they used in their works for others to reproduce and validate their methods, as sharing those data directly can lead to significant privacy risks (e.g., against inference attacks). To solve the problem and expedite cooperative scientific research, we propose a novel differentially private sharing mechanism for genomic datasets that protects the entire genomic dataset under differential privacy. To improve data utility of the GWAS statistics, we further develop a post-processing scheme that performs optimal transport (OT) on the empirical distributions of SNP values. The distributions are also achieved in a privacy-preserving manner. We evaluate our approach on several real genomic datasets and show in the experiments that it provides better protection against both genomic and machine learning-based membership inference attacks and offers higher GWAS utility than the baseline approaches.
[[2209.06529] Data Privacy and Trustworthy Machine Learning](http://arxiv.org/abs/2209.06529)The privacy risks of machine learning models is a major concern when training them on sensitive and personal data. We discuss the tradeoffs between data privacy and the remaining goals of trustworthy machine learning (notably, fairness, robustness, and explainability).
[[2209.06556] Cryptanalysis of a privacy-preserving behavior-oriented authentication scheme](http://arxiv.org/abs/2209.06556)Continuous authentication has been proposed as a complementary security mechanism to password-based authentication for computer devices that are handled directly by humans, such as smart phones. Continuous authentication has some privacy issues as certain user features and actions are revealed to the authentication server, which is not assumed to be trusted. Wei et al. proposed in 2021 a privacy-preserving protocol for behavioral authentication that utilizes homomorphic encryption. The encryption prevents the server from obtaining sampled user features. In this paper, we show that the Wei et al. scheme is insecure regarding both an honest-but-curious server and an active eavesdropper. We present two attacks: The first attack enables the authentication server to obtain the secret user key, plaintext behavior template and plaintext authentication behavior data from encrypted data. The second attack enables an active eavesdropper to restore the plaintext authentication behavior data from the transmitted encrypted data.
[[2209.06557] A Generic Privacy-Preserving Protocol For Keystroke Dynamics-Based Continuous Authentication](http://arxiv.org/abs/2209.06557)Continuous authentication utilizes automatic recognition of certain user features for seamless and passive authentication without requiring user attention. Such features can be divided into categories of physiological biometrics and behavioral biometrics. Keystroke dynamics is proposed for behavioral biometrics-oriented authentication by recognizing users by means of their typing patterns. However, it has been pointed out that continuous authentication using physiological biometrics and behavior biometrics incur privacy risks, revealing personal characteristics and activities. In this paper, we consider a previously proposed keystroke dynamics-based authentication scheme that has no privacy-preserving properties. In this regard, we propose a generic privacy-preserving version of this authentication scheme in which all user features are encrypted -- preventing disclosure of those to the authentication server. Our scheme is generic in the sense that it assumes homomorphic cryptographic primitives. Authentication is conducted on the basis of encrypted data due to the homomorphic cryptographic properties of our protocol.
[[2209.06300] PINCH: An Adversarial Extraction Attack Framework for Deep Learning Models](http://arxiv.org/abs/2209.06300)Deep Learning (DL) models increasingly power a diversity of applications. Unfortunately, this pervasiveness also makes them attractive targets for extraction attacks which can steal the architecture, parameters, and hyper-parameters of a targeted DL model. Existing extraction attack studies have observed varying levels of attack success for different DL models and datasets, yet the underlying cause(s) behind their susceptibility often remain unclear. Ascertaining such root-cause weaknesses would help facilitate secure DL systems, though this requires studying extraction attacks in a wide variety of scenarios to identify commonalities across attack success and DL characteristics. The overwhelmingly high technical effort and time required to understand, implement, and evaluate even a single attack makes it infeasible to explore the large number of unique extraction attack scenarios in existence, with current frameworks typically designed to only operate for specific attack types, datasets and hardware platforms. In this paper we present PINCH: an efficient and automated extraction attack framework capable of deploying and evaluating multiple DL models and attacks across heterogeneous hardware platforms. We demonstrate the effectiveness of PINCH by empirically evaluating a large number of previously unexplored extraction attack scenarios, as well as secondary attack staging. Our key findings show that 1) multiple characteristics affect extraction attack success spanning DL model architecture, dataset complexity, hardware, attack type, and 2) partially successful extraction attacks significantly enhance the success of further adversarial attack staging.
[[2209.06388] TSFool: Crafting High-quality Adversarial Time Series through Multi-objective Optimization to Fool Recurrent Neural Network Classifiers](http://arxiv.org/abs/2209.06388)Deep neural network (DNN) classifiers are vulnerable to adversarial attacks. Although the existing gradient-based attacks have achieved good performance in feed-forward model and image recognition tasks, the extension for time series classification in the recurrent neural network (RNN) remains a dilemma, because the cyclical structure of RNN prevents direct model differentiation and the visual sensitivity to perturbations of time series data challenges the traditional local optimization objective to minimize perturbation. In this paper, an efficient and widely applicable approach called TSFool for crafting high-quality adversarial time series for the RNN classifier is proposed. We propose a novel global optimization objective named Camouflage Coefficient to consider how well the adversarial samples hide in class clusters, and accordingly redefine the high-quality adversarial attack as a multi-objective optimization problem. We also propose a new idea to use intervalized weighted finite automata (IWFA) to capture deeply embedded vulnerable samples having otherness between features and latent manifold to guide the approximation to the optimization solution. Experiments on 22 UCR datasets are conducted to confirm that TSFool is a widely effective, efficient and high-quality approach with 93.22% less local perturbation, 32.33% better global camouflage, and 1.12 times speedup to existing methods.
[[2209.06397] Federated Learning based on Defending Against Data Poisoning Attacks in IoT](http://arxiv.org/abs/2209.06397)The rapidly expanding number of Internet of Things (IoT) devices is generating huge quantities of data, but the data privacy and security exposure in IoT devices, especially in the automatic driving system. Federated learning (FL) is a paradigm that addresses data privacy, security, access rights, and access to heterogeneous message issues by integrating a global model based on distributed nodes. However, data poisoning attacks on FL can undermine the benefits, destroying the global model's availability and disrupting model training. To avoid the above issues, we build up a hierarchical defense data poisoning (HDDP) system framework to defend against data poisoning attacks in FL, which monitors each local model of individual nodes via abnormal detection to remove the malicious clients. Whether the poisoning defense server has a trusted test dataset, we design the \underline{l}ocal \underline{m}odel \underline{t}est \underline{v}oting (LMTV) and \underline{k}ullback-\underline{l}eibler divergence \underline{a}nomaly parameters \underline{d}etection (KLAD) algorithms to defend against label-flipping poisoning attacks. Specifically, the trusted test dataset is utilized to obtain the evaluation results for each classification to recognize the malicious clients in LMTV. More importantly, we adopt the kullback leibler divergence to measure the similarity between local models without the trusted test dataset in KLAD. Finally, through extensive evaluations and against the various label-flipping poisoning attacks, LMTV and KLAD algorithms could achieve the $100\%$ and $40\%$ to $85\%$ successful defense ratios under different detection situations.
[[2209.06722] Detection of Smart Grid Integrity Attacks Using Signal Temporal Logic](http://arxiv.org/abs/2209.06722)Cyber-attacks can have severe impacts on critical infrastructures, from outages to economical loss and physical damage to people and environment. One of the main targets of these attacks is the smart grid. In this paper, we propose a new software detector for integrity attacks targeting smart meter readings. The detector relies upon mining parameters of temporal logic specifications for integrity attack classification. To this end, we use Signal Temporal Logic (STL) for specifying properties over time series. Our approach considers different "attack scenarios" found in last years: given a parametric formula for each "attack scenario" and a set of labeled traces, we aim at finding the parameter valuation that validates each template.
[[2209.06322] FaceTopoNet: Facial Expression Recognition using Face Topology Learning](http://arxiv.org/abs/2209.06322)Prior work has shown that the order in which different components of the face are learned using a sequential learner can play an important role in the performance of facial expression recognition systems. We propose FaceTopoNet, an end-to-end deep model for facial expression recognition, which is capable of learning an effective tree topology of the face. Our model then traverses the learned tree to generate a sequence, which is then used to form an embedding to feed a sequential learner. The devised model adopts one stream for learning structure and one stream for learning texture. The structure stream focuses on the positions of the facial landmarks, while the main focus of the texture stream is on the patches around the landmarks to learn textural information. We then fuse the outputs of the two streams by utilizing an effective attention-based fusion strategy. We perform extensive experiments on four large-scale in-the-wild facial expression datasets - namely AffectNet, FER2013, ExpW, and RAF-DB - and one lab-controlled dataset (CK+) to evaluate our approach. FaceTopoNet achieves state-of-the-art performance on three of the five datasets and obtains competitive results on the other two datasets. We also perform rigorous ablation and sensitivity experiments to evaluate the impact of different components and parameters in our model. Lastly, we perform robustness experiments and demonstrate that FaceTopoNet is more robust against occlusions in comparison to other leading methods in the area.
[[2209.06376] iSimLoc: Visual Global Localization for Previously Unseen Environments with Simulated Images](http://arxiv.org/abs/2209.06376)The visual camera is an attractive device in beyond visual line of sight (B-VLOS) drone operation, since they are low in size, weight, power, and cost, and can provide redundant modality to GPS failures. However, state-of-the-art visual localization algorithms are unable to match visual data that have a significantly different appearance due to illuminations or viewpoints. This paper presents iSimLoc, a condition/viewpoint consistent hierarchical global re-localization approach. The place features of iSimLoc can be utilized to search target images under changing appearances and viewpoints. Additionally, our hierarchical global re-localization module refines in a coarse-to-fine manner, allowing iSimLoc to perform a fast and accurate estimation. We evaluate our method on one dataset with appearance variations and one dataset that focuses on demonstrating large-scale matching over a long flight in complicated environments. On our two datasets, iSimLoc achieves 88.7\% and 83.8\% successful retrieval rates with 1.5s inferencing time, compared to 45.8% and 39.7% using the next best method. These results demonstrate robust localization in a range of environments.
[[2209.06395] Point Cloud Registration-Driven Robust Feature Matching for 3D Siamese Object Tracking](http://arxiv.org/abs/2209.06395)Learning robust feature matching between the template and search area is crucial for 3D Siamese tracking. The core of Siamese feature matching is how to assign high feature similarity on the corresponding points between the template and search area for precise object localization. In this paper, we propose a novel point cloud registration-driven Siamese tracking framework, with the intuition that spatially aligned corresponding points (via 3D registration) tend to achieve consistent feature representations. Specifically, our method consists of two modules, including a tracking-specific nonlocal registration module and a registration-aided Sinkhorn template-feature aggregation module. The registration module targets at the precise spatial alignment between the template and search area. The tracking-specific spatial distance constraint is proposed to refine the cross-attention weights in the nonlocal module for discriminative feature learning. Then, we use the weighted SVD to compute the rigid transformation between the template and search area, and align them to achieve the desired spatially aligned corresponding points. For the feature aggregation model, we formulate the feature matching between the transformed template and search area as an optimal transport problem and utilize the Sinkhorn optimization to search for the outlier-robust matching solution. Also, a registration-aided spatial distance map is built to improve the matching robustness in indistinguishable regions (e.g., smooth surface). Finally, guided by the obtained feature matching map, we aggregate the target information from the template into the search area to construct the target-specific feature, which is then fed into a CenterPoint-like detection head for object localization. Extensive experiments on KITTI, NuScenes and Waymo datasets verify the effectiveness of our proposed method.
[[2209.06423] SCULPTOR: Skeleton-Consistent Face Creation Using a Learned Parametric Generator](http://arxiv.org/abs/2209.06423)Recent years have seen growing interest in 3D human faces modelling due to its wide applications in digital human, character generation and animation. Existing approaches overwhelmingly emphasized on modeling the exterior shapes, textures and skin properties of faces, ignoring the inherent correlation between inner skeletal structures and appearance. In this paper, we present SCULPTOR, 3D face creations with Skeleton Consistency Using a Learned Parametric facial generaTOR, aiming to facilitate easy creation of both anatomically correct and visually convincing face models via a hybrid parametric-physical representation. At the core of SCULPTOR is LUCY, the first large-scale shape-skeleton face dataset in collaboration with plastic surgeons. Named after the fossils of one of the oldest known human ancestors, our LUCY dataset contains high-quality Computed Tomography (CT) scans of the complete human head before and after orthognathic surgeries, critical for evaluating surgery results. LUCY consists of 144 scans of 72 subjects (31 male and 41 female) where each subject has two CT scans taken pre- and post-orthognathic operations. Based on our LUCY dataset, we learn a novel skeleton consistent parametric facial generator, SCULPTOR, which can create the unique and nuanced facial features that help define a character and at the same time maintain physiological soundness. Our SCULPTOR jointly models the skull, face geometry and face appearance under a unified data-driven framework, by separating the depiction of a 3D face into shape blend shape, pose blend shape and facial expression blend shape. SCULPTOR preserves both anatomic correctness and visual realism in facial generation tasks compared with existing methods. Finally, we showcase the robustness and effectiveness of SCULPTOR in various fancy applications unseen before.
[[2209.06535] CRAFT: Camera-Radar 3D Object Detection with Spatio-Contextual Fusion Transformer](http://arxiv.org/abs/2209.06535)Camera and radar sensors have significant advantages in cost, reliability, and maintenance compared to LiDAR. Existing fusion methods often fuse the outputs of single modalities at the result-level, called the late fusion strategy. This can benefit from using off-the-shelf single sensor detection algorithms, but late fusion cannot fully exploit the complementary properties of sensors, thus having limited performance despite the huge potential of camera-radar fusion. Here we propose a novel proposal-level early fusion approach that effectively exploits both spatial and contextual properties of camera and radar for 3D object detection. Our fusion framework first associates image proposal with radar points in the polar coordinate system to efficiently handle the discrepancy between the coordinate system and spatial properties. Using this as a first stage, following consecutive cross-attention based feature fusion layers adaptively exchange spatio-contextual information between camera and radar, leading to a robust and attentive fusion. Our camera-radar fusion approach achieves the state-of-the-art 41.1% mAP and 52.3% NDS on the nuScenes test set, which is 8.7 and 10.8 points higher than the camera-only baseline, as well as yielding competitive performance on the LiDAR method.
[[2209.06562] INV-Flow2PoseNet: Light-Resistant Rigid Object Pose from Optical Flow of RGB-D Images using Images, Normals and Vertices](http://arxiv.org/abs/2209.06562)This paper presents a novel architecture for simultaneous estimation of highly accurate optical flows and rigid scene transformations for difficult scenarios where the brightness assumption is violated by strong shading changes. In the case of rotating objects or moving light sources, such as those encountered for driving cars in the dark, the scene appearance often changes significantly from one view to the next. Unfortunately, standard methods for calculating optical flows or poses are based on the expectation that the appearance of features in the scene remain constant between views. These methods may fail frequently in the investigated cases. The presented method fuses texture and geometry information by combining image, vertex and normal data to compute an illumination-invariant optical flow. By using a coarse-to-fine strategy, globally anchored optical flows are learned, reducing the impact of erroneous shading-based pseudo-correspondences. Based on the learned optical flows, a second architecture is proposed that predicts robust rigid transformations from the warped vertex and normal maps. Particular attention is payed to situations with strong rotations, which often cause such shading changes. Therefore a 3-step procedure is proposed that profitably exploits correlations between the normals and vertices. The method has been evaluated on a newly created dataset containing both synthetic and real data with strong rotations and shading effects. This data represents the typical use case in 3D reconstruction, where the object often rotates in large steps between the partial reconstructions. Additionally, we apply the method to the well-known Kitti Odometry dataset. Even if, due to fulfillment of the brighness assumption, this is not the typical use case of the method, the applicability to standard situations and the relation to other methods is therefore established.
[[2209.06681] A Benchmark and a Baseline for Robust Multi-view Depth Estimation](http://arxiv.org/abs/2209.06681)Recent deep learning approaches for multi-view depth estimation are employed either in a depth-from-video or a multi-view stereo setting. Despite different settings, these approaches are technically similar: they correlate multiple source views with a keyview to estimate a depth map for the keyview. In this work, we introduce the Robust Multi-View Depth Benchmark that is built upon a set of public datasets and allows evaluation in both settings on data from different domains. We evaluate recent approaches and find imbalanced performances across domains. Further, we consider a third setting, where camera poses are available and the objective is to estimate the corresponding depth maps with their correct scale. We show that recent approaches do not generalize across datasets in this setting. This is because their cost volume output runs out of distribution. To resolve this, we present the Robust MVD Baseline model for multi-view depth estimation, which is built upon existing components but employs a novel scale augmentation procedure. It can be applied for robust multi-view depth estimation, independent of the target data. We provide code for the proposed benchmark and baseline model at https://github.com/lmb-freiburg/robustmvd.
[[2209.06717] Out-of-Vocabulary Challenge Report](http://arxiv.org/abs/2209.06717)This paper presents final results of the Out-Of-Vocabulary 2022 (OOV) challenge. The OOV contest introduces an important aspect that is not commonly studied by Optical Character Recognition (OCR) models, namely, the recognition of unseen scene text instances at training time. The competition compiles a collection of public scene text datasets comprising of 326,385 images with 4,864,405 scene text instances, thus covering a wide range of data distributions. A new and independent validation and test set is formed with scene text instances that are out of vocabulary at training time. The competition was structured in two tasks, end-to-end and cropped scene text recognition respectively. A thorough analysis of results from baselines and different participants is presented. Interestingly, current state-of-the-art models show a significant performance gap under the newly studied setting. We conclude that the OOV dataset proposed in this challenge will be an essential area to be explored in order to develop scene text models that achieve more robust and generalized predictions.
[[2209.06442] SUN: Exploring Intrinsic Uncertainties in Text-to-SQL Parsers](http://arxiv.org/abs/2209.06442)This paper aims to improve the performance of text-to-SQL parsing by exploring the intrinsic uncertainties in the neural network based approaches (called SUN). From the data uncertainty perspective, it is indisputable that a single SQL can be learned from multiple semantically-equivalent questions.Different from previous methods that are limited to one-to-one mapping, we propose a data uncertainty constraint to explore the underlying complementary semantic information among multiple semantically-equivalent questions (many-to-one) and learn the robust feature representations with reduced spurious associations. In this way, we can reduce the sensitivity of the learned representations and improve the robustness of the parser. From the model uncertainty perspective, there is often structural information (dependence) among the weights of neural networks. To improve the generalizability and stability of neural text-to-SQL parsers, we propose a model uncertainty constraint to refine the query representations by enforcing the output representations of different perturbed encoding networks to be consistent with each other. Extensive experiments on five benchmark datasets demonstrate that our method significantly outperforms strong competitors and achieves new state-of-the-art results. For reproducibility, we release our code and data at https://github.com/AlibabaResearch/DAMO-ConvAI/tree/main/sunsql.
[[2209.06517] How to Find Strong Summary Coherence Measures? A Toolbox and a Comparative Study for Summary Coherence Measure Evaluation](http://arxiv.org/abs/2209.06517)Automatically evaluating the coherence of summaries is of great significance both to enable cost-efficient summarizer evaluation and as a tool for improving coherence by selecting high-scoring candidate summaries. While many different approaches have been suggested to model summary coherence, they are often evaluated using disparate datasets and metrics. This makes it difficult to understand their relative performance and identify ways forward towards better summary coherence modelling. In this work, we conduct a large-scale investigation of various methods for summary coherence modelling on an even playing field. Additionally, we introduce two novel analysis measures, intra-system correlation and bias matrices, that help identify biases in coherence measures and provide robustness against system-level confounders. While none of the currently available automatic coherence measures are able to assign reliable coherence scores to system summaries across all evaluation metrics, large-scale language models fine-tuned on self-supervised tasks show promising results, as long as fine-tuning takes into account that they need to generalize across different summary lengths.
[[2209.06596] Few Clean Instances Help Denoising Distant Supervision](http://arxiv.org/abs/2209.06596)Existing distantly supervised relation extractors usually rely on noisy data for both model training and evaluation, which may lead to garbage-in-garbage-out systems. To alleviate the problem, we study whether a small clean dataset could help improve the quality of distantly supervised models. We show that besides getting a more convincing evaluation of models, a small clean dataset also helps us to build more robust denoising models. Specifically, we propose a new criterion for clean instance selection based on influence functions. It collects sample-level evidence for recognizing good instances (which is more informative than loss-level evidence). We also propose a teacher-student mechanism for controlling purity of intermediate results when bootstrapping the clean set. The whole approach is model-agnostic and demonstrates strong performances on both denoising real (NYT) and synthetic noisy datasets.
[[2209.06701] Natural Language Inference Prompts for Zero-shot Emotion Classification in Text across Corpora](http://arxiv.org/abs/2209.06701)Within textual emotion classification, the set of relevant labels depends on the domain and application scenario and might not be known at the time of model development. This conflicts with the classical paradigm of supervised learning in which the labels need to be predefined. A solution to obtain a model with a flexible set of labels is to use the paradigm of zero-shot learning as a natural language inference task, which in addition adds the advantage of not needing any labeled training data. This raises the question how to prompt a natural language inference model for zero-shot learning emotion classification. Options for prompt formulations include the emotion name anger alone or the statement "This text expresses anger". With this paper, we analyze how sensitive a natural language inference-based zero-shot-learning classifier is to such changes to the prompt under consideration of the corpus: How carefully does the prompt need to be selected? We perform experiments on an established set of emotion datasets presenting different language registers according to different sources (tweets, events, blogs) with three natural language inference models and show that indeed the choice of a particular prompt formulation needs to fit to the corpus. We show that this challenge can be tackled with combinations of multiple prompts. Such ensemble is more robust across corpora than individual prompts and shows nearly the same performance as the individual best prompt for a particular corpus.
[[2209.06720] On Language Clustering: A Non-parametric Statistical Approach](http://arxiv.org/abs/2209.06720)Any approach aimed at pasteurizing and quantifying a particular phenomenon must include the use of robust statistical methodologies for data analysis. With this in mind, the purpose of this study is to present statistical approaches that may be employed in nonparametric nonhomogeneous data frameworks, as well as to examine their application in the field of natural language processing and language clustering. Furthermore, this paper discusses the many uses of nonparametric approaches in linguistic data mining and processing. The data depth idea allows for the centre-outward ordering of points in any dimension, resulting in a new nonparametric multivariate statistical analysis that does not require any distributional assumptions. The concept of hierarchy is used in historical language categorisation and structuring, and it aims to organise and cluster languages into subfamilies using the same premise. In this regard, the current study presents a novel approach to language family structuring based on non-parametric approaches produced from a typological structure of words in various languages, which is then converted into a Cartesian framework using MDS. This statistical-depth-based architecture allows for the use of data-depth-based methodologies for robust outlier detection, which is extremely useful in understanding the categorization of diverse borderline languages and allows for the re-evaluation of existing classification systems. Other depth-based approaches are also applied to processes such as unsupervised and supervised clustering. This paper therefore provides an overview of procedures that can be applied to nonhomogeneous language classification systems in a nonparametric framework.
[[2209.06767] Parameter-Efficient Finetuning for Robust Continual Multilingual Learning](http://arxiv.org/abs/2209.06767)NLU systems deployed in the real world are expected to be regularly updated by retraining or finetuning the underlying neural network on new training examples accumulated over time. In our work, we focus on the multilingual setting where we would want to further finetune a multilingual model on new training data for the same NLU task on which the aforementioned model has already been trained for. We show that under certain conditions, naively updating the multilingual model can lead to losses in performance over a subset of languages although the aggregated performance metric shows an improvement. We establish this phenomenon over four tasks belonging to three task families (token-level, sentence-level and seq2seq) and find that the baseline is far from ideal for the setting at hand. We then build upon recent advances in parameter-efficient finetuning to develop novel finetuning pipelines that allow us to jointly minimize catastrophic forgetting while encouraging positive cross-lingual transfer, hence improving the spread of gains over different languages while reducing the losses incurred in this setup.
[[2209.06259] Designing Biological Sequences via Meta-Reinforcement Learning and Bayesian Optimization](http://arxiv.org/abs/2209.06259)The ability to accelerate the design of biological sequences can have a substantial impact on the progress of the medical field. The problem can be framed as a global optimization problem where the objective is an expensive black-box function such that we can query large batches restricted with a limitation of a low number of rounds. Bayesian Optimization is a principled method for tackling this problem. However, the astronomically large state space of biological sequences renders brute-force iterating over all possible sequences infeasible. In this paper, we propose MetaRLBO where we train an autoregressive generative model via Meta-Reinforcement Learning to propose promising sequences for selection via Bayesian Optimization. We pose this problem as that of finding an optimal policy over a distribution of MDPs induced by sampling subsets of the data acquired in the previous rounds. Our in-silico experiments show that meta-learning over such ensembles provides robustness against reward misspecification and achieves competitive results compared to existing strong baselines.
[[2209.06620] Distributionally Robust Offline Reinforcement Learning with Linear Function Approximation](http://arxiv.org/abs/2209.06620)Among the reasons that hinder the application of reinforcement learning (RL) to real-world problems, two factors are critical: limited data and the mismatch of the testing environment compared to training one. In this paper, we attempt to address these issues simultaneously with the problem setup of distributionally robust offline RL. Particularly, we learn an RL agent with the historical data obtained from the source environment and optimize it to perform well in the perturbed one. Moreover, we consider the linear function approximation to apply the algorithm to large-scale problems. We prove our algorithm can achieve the suboptimality of $O(1/\sqrt{K})$ depending on the linear function dimension $d$, which seems to be the first result with sample complexity guarantee in this setting. Diverse experiments are conducted to demonstrate our theoretical findings, showing the superiority of our algorithm against the non-robust one.
[[2209.06257] SciMED: A Computational Framework For Physics-Informed Symbolic Regression with Scientist-In-The-Loop](http://arxiv.org/abs/2209.06257)Discovering a meaningful, dimensionally homogeneous, symbolic expression that explains experimental data is a fundamental challenge in many scientific fields. We present a novel, open-source computational framework called Scientist-Machine Equation Detector (SciMED), which integrates scientific discipline wisdom in a scientist-in-the-loop approach with state-of-the-art symbolic regression (SR) methods. SciMED combines a genetic algorithm-based wrapper selection method with automatic machine learning and two levels of SR methods. We test SciMED on four configurations of the settling of a sphere with and without a non-linear aerodynamic drag force. We show that SciMED is sufficiently robust to discover the correct physically meaningful symbolic expressions from noisy data. Our results indicate better performance on these tasks than the state-of-the-art SR software package.
[[2209.06428] Semantic Visual Simultaneous Localization and Mapping: A Survey](http://arxiv.org/abs/2209.06428)Visual Simultaneous Localization and Mapping (vSLAM) has achieved great progress in the computer vision and robotics communities, and has been successfully used in many fields such as autonomous robot navigation and AR/VR. However, vSLAM cannot achieve good localization in dynamic and complex environments. Numerous publications have reported that, by combining with the semantic information with vSLAM, the semantic vSLAM systems have the capability of solving the above problems in recent years. Nevertheless, there is no comprehensive survey about semantic vSLAM. To fill the gap, this paper first reviews the development of semantic vSLAM, explicitly focusing on its strengths and differences. Secondly, we explore three main issues of semantic vSLAM: the extraction and association of semantic information, the application of semantic information, and the advantages of semantic vSLAM. Then, we collect and analyze the current state-of-the-art SLAM datasets which have been widely used in semantic vSLAM systems. Finally, we discuss future directions that will provide a blueprint for the future development of semantic vSLAM.
[[2209.06515] Learning to Evaluate Performance of Multi-modal Semantic Localization](http://arxiv.org/abs/2209.06515)Semantic localization (SeLo) refers to the task of obtaining the most relevant locations in large-scale remote sensing (RS) images using semantic information such as text. As an emerging task based on cross-modal retrieval, SeLo achieves semantic-level retrieval with only caption-level annotation, which demonstrates its great potential in unifying downstream tasks. Although SeLo has been carried out successively, but there is currently no work has systematically explores and analyzes this urgent direction. In this paper, we thoroughly study this field and provide a complete benchmark in terms of metrics and testdata to advance the SeLo task. Firstly, based on the characteristics of this task, we propose multiple discriminative evaluation metrics to quantify the performance of the SeLo task. The devised significant area proportion, attention shift distance, and discrete attention distance are utilized to evaluate the generated SeLo map from pixel-level and region-level. Next, to provide standard evaluation data for the SeLo task, we contribute a diverse, multi-semantic, multi-objective Semantic Localization Testset (AIR-SLT). AIR-SLT consists of 22 large-scale RS images and 59 test cases with different semantics, which aims to provide a comprehensive evaluations for retrieval models. Finally, we analyze the SeLo performance of RS cross-modal retrieval models in detail, explore the impact of different variables on this task, and provide a complete benchmark for the SeLo task. We have also established a new paradigm for RS referring expression comprehension, and demonstrated the great advantage of SeLo in semantics through combining it with tasks such as detection and road extraction. The proposed evaluation metrics, semantic localization testsets, and corresponding scripts have been open to access at github.com/xiaoyuan1996/SemanticLocalizationMetrics .
[[2209.06210] Scheduling Algorithms for Federated Learning with Minimal Energy Consumption](http://arxiv.org/abs/2209.06210)Federated Learning (FL) has opened the opportunity for collaboratively training machine learning models on heterogeneous mobile or Edge devices while keeping local data private.With an increase in its adoption, a growing concern is related to its economic and environmental cost (as is also the case for other machine learning techniques).Unfortunately, little work has been done to optimize its energy consumption or emissions of carbon dioxide or equivalents, as energy minimization is usually left as a secondary objective.In this paper, we investigate the problem of minimizing the energy consumption of FL training on heterogeneous devices by controlling the workload distribution.We model this as the Minimal Cost FL Schedule problem, a total cost minimization problem with identical, independent, and atomic tasks that have to be assigned to heterogeneous resources with arbitrary cost functions.We propose a pseudo-polynomial optimal solution to the problem based on the previously unexplored Multiple-Choice Minimum-Cost Maximal Knapsack Packing Problem.We also provide four algorithms for scenarios where cost functions are monotonically increasing and follow the same behavior.These solutions are likewise applicable on the minimization of other kinds of costs, and in other one-dimensional data partition problems.
[[2209.06359] Federated Pruning: Improving Neural Network Efficiency with Federated Learning](http://arxiv.org/abs/2209.06359)Automatic Speech Recognition models require large amount of speech data for training, and the collection of such data often leads to privacy concerns. Federated learning has been widely used and is considered to be an effective decentralized technique by collaboratively learning a shared prediction model while keeping the data local on different clients devices. However, the limited computation and communication resources on clients devices present practical difficulties for large models. To overcome such challenges, we propose Federated Pruning to train a reduced model under the federated setting, while maintaining similar performance compared to the full model. Moreover, the vast amount of clients data can also be leveraged to improve the pruning results compared to centralized training. We explore different pruning schemes and provide empirical evidence of the effectiveness of our methods.
[[2209.06623] Age of Information in Federated Learning over Wireless Networks](http://arxiv.org/abs/2209.06623)In this paper, federated learning (FL) over wireless networks is investigated. In each communication round, a subset of devices is selected to participate in the aggregation with limited time and energy. In order to minimize the convergence time, global loss and latency are jointly considered in a Stackelberg game based framework. Specifically, age of information (AoI) based device selection is considered at leader-level as a global loss minimization problem, while sub-channel assignment, computational resource allocation, and power allocation are considered at follower-level as a latency minimization problem. By dividing the follower-level problem into two sub-problems, the best response of the follower is obtained by a monotonic optimization based resource allocation algorithm and a matching based sub-channel assignment algorithm. By deriving the upper bound of convergence rate, the leader-level problem is reformulated, and then a list based device selection algorithm is proposed to achieve Stackelberg equilibrium. Simulation results indicate that the proposed device selection scheme outperforms other schemes in terms of the global loss, and the developed algorithms can significantly decrease the time consumption of computation and communication.