The traditional method of data management at the MRC Clinical Trials Unit has been for paper CRFs to be completed by accredited sites and sent in to a coordinating data centre. There the data is entered on to a trial database and cleaned. However there has been a shift recently within our unit towards electronic data capture, whereby sites enter data directly into the trial database. Data monit...

Data cleaning has become an indispensable part of data analysis due to the increasing amount of dirty data. Data scientists spend most of their time preparing dirty data before it can be used for data analysis. At the same time, the existing tools that attempt to automate the data cleaning procedure typically focus on a specific use case and operation. Still, even such specialized tools exhibit...

In many published articles, there is still no mention of quality control processes, which might be an indication of the insufficient importance the researchers attach to undertaking or reporting such processes. However, quality control of data is one of the most important steps in research projects. Lack of sufficient attention to quality control of data might have a detrimental effect on the r...

There is no magic solution for data cleaning. The user has always to specify the cleaning operations to perform. A huge number of operations may have to be specified. Yet, this is the condition to detect and correct the data quality problems successfully. Most of the cleaning operations are generic enough to be applied to different databases. These operations may be limited to databases of the ...

Knowledge discovery is an important part of reservoir management, and it is also a bottleneck of widespread application of knowledge. So we should make use of some particular data mining methods to discover knowledge, which should be based on some particular fields. According to demand of oil and gas development and characters of reservoir data set, puts forward a new idea of knowledge discover...

BACKGROUND Within the field of record linkage, numerous data cleaning and standardisation techniques are employed to ensure the highest quality of links. While these facilities are common in record linkage software packages and are regularly deployed across record linkage units, little work has been published demonstrating the impact of data cleaning on linkage quality. METHODS A range of cle...

Declarative rules, such as functional dependencies, are widely used for cleaning data. Several systems take them as input for detecting errors and computing a “clean” version of the data. To support domain experts,in specifying these rules, several tools have been proposed to profile the data and mine rules. However, existing discovery techniques have traditionally ignored the time dimension. R...