Downloads & Free Reading Options - Results

A brief overview of current status of data compression technologies are presented with the interim development of compression software and tests of effectiveness on the selected digitized data types. Several algorithms such as Rice, LZ-family, and run-length codings are examined for selected digitized packet data streams. These algorithms are able to compress most data available with a fairly good compression factor except for Tape A and encrypted data. Tape A data contains a mixture of several data formats in very short runs (several packets), rather than the long runs (typically hundreds of packets), for which the current version of the software was designed. The remaining data sets were compressed with a compression factor of over 2.74.

This article discusses the basic methods of encoding video data and their shortcomings. It describes the problem of distortion waveforms of video when encoding in the study of test frames of the primary array of video data. 

This report summarises the results of an investigation of the applicability of the JPEG20O0 compression system to Digital Terrain Elevation Data (DTED) and in particular the application of the region-of-interest, ROI, capabilities built into the standard. JPEG2000 is a new image compression standard offering a number of advantages, particularly relevant to Defence, over previous image compression standards. Its application to DTED would not normally be considered, given that DTED is not an imagery data type. However DTED has similar characteristics to monochrome imagery data and the utility of JPEG2O00 and in particular ROI capability would have benefits for the storage and dissemination of DTED within Defence. The advantage of ROI coding is that very high compression ratios can be achieved while maintaining high fidelity of the data in regions of importance. Possible applications of such techniques would be in the distribution or dissemination of DTED, or mission rehearsal products containing elevation data, to deployed units where they could be used to support tactical mission planning and rehearsal.

The report demonstrates the applicability of classical statistical techniques to problems involving compression and classification of multivariate data. The theoretical foundations of two such techniques, intrinsic analysis and discriminant analysis, are treated in detail. Efficient digital computer implementation is discussed, including the combined application of intrinsic and discriminant analysis and a new algorithm for computing approximate intrinsic bases for very large problems. Experimental results are presented on the application of these techniques as feature extractors in a signal classification problem. Also included is a description of the interactive graphics-oriented system software which has been developed to facilitate the application of these techniques.

Huffman compression is a statistical, lossless, data compression algorithm that compresses data by assigning variable length codes to symbols, with the more frequently appearing symbols given shorter codes than the less. This work is a modification of the Huffman algorithm which permits uncompressed data to be decomposed into indepen- dently compressible and decompressible blocks, allowing for concurrent compression and decompression on multiple processors. We create implementations of this modified algorithm on a current NVIDIA GPU using the CUDA API as well as on a current Intel chip and the performance results are compared, showing favorable GPU performance for nearly all tests. Lastly, we discuss the necessity for high performance data compression in today's supercomputing ecosystem.

End-to-end system considerations involving channel coding and data compression are reported which could drastically improve the efficiency in communicating pictorial information from future planetary spacecraft. In addition to presenting new and potentially significant system considerations, this report attempts to fill a need for a comprehensive tutorial which makes much of this very subject accessible to readers whose disciplines lie outside of communication theory.

Efficient onboard data compression can reduce the data volume from hyperspectral imagers on NASA and DoD spacecraft in order to return as much imagery as possible through constrained downlink channels. Lossless compression is important for signature extraction, object recognition, and feature classification capabilities. To provide onboard data compression, a hardware implementation of a lossless hyperspectral compression algorithm was developed using a field programmable gate array (FPGA). The underlying algorithm is the Fast Lossless (FL) compression algorithm reported in Fast Lossless Compression of Multispectral- Image Data (NPO-42517), NASA Tech Briefs, Vol. 30, No. 8 (August 2006), p. 26 with the modification reported in Lossless, Multi-Spectral Data Comressor for Improved Compression for Pushbroom-Type Instruments (NPO-45473), NASA Tech Briefs, Vol. 32, No. 7 (July 2008) p. 63, which provides improved compression performance for data from pushbroom-type imagers. An FPGA implementation of the unmodified FL algorithm was previously developed and reported in Fast and Adaptive Lossless Onboard Hyperspectral Data Compression System (NPO-46867), NASA Tech Briefs, Vol. 36, No. 5 (May 2012) p. 42. The essence of the FL algorithm is adaptive linear predictive compression using the sign algorithm for filter adaption. The FL compressor achieves a combination of low complexity and compression effectiveness that exceeds that of stateof- the-art techniques currently in use. The modification changes the predictor structure to tolerate differences in sensitivity of different detector elements, as occurs in pushbroom-type imagers, which are suitable for spacecraft use. The FPGA implementation offers a low-cost, flexible solution compared to traditional ASIC (application specific integrated circuit) and can be integrated as an intellectual property (IP) for part of, e.g., a design that manages the instrument interface. The FPGA implementation was benchmarked on the Xilinx Virtex IV LX25 device, and ported to a Xilinx prototype board. The current implementation has a critical path of 29.5 ns, which dictated a clock speed of 33 MHz. The critical path delay is end-to-end measurement between the uncompressed input data and the output compression data stream. The implementation compresses one sample every clock cycle, which results in a speed of 33 Msample/s. The implementation has a rather low device use of the Xilinx Virtex IV LX25, making the total power consumption of the implementation about 1.27 W.

The objectives of this study were to determine the range of validity of a Dynamic Digital Blade Row Compression Component Stability Computer Model by comparing the results of model simulations with previously obtained test data and to determine the manner in which a fan component responded to imposed unsteady aerodynamics by examining calculated parameters. Validation of the modeling technique was obtained by simulating a facility and test conditions where the response of a two-stage fan to planar waves was determined. The simulation of the two-stage fan component was carried out using a quasi one-dimensional, pitch line, blade-row by blade-row model which used blade-row loss coefficients and deviation angles determined from analysis of clean-inlet-flow test data. This model was ideally suited for studying the fan response to the planar waves produced by the Planar Pressure Pulse Generator, a device which produced highly planar waves with peak-to-peak fluctuations of 10-30 percent of the mean total pressure over a frequency range of 42 to 800 Hz. The results of the simulations showed that the model accurately reproduced the waveshape, amplitude, and phase of the experimentally determined fan inlet and exit pressures and the fan inlet flow up to 75 Hz where the need for including unsteady blade circulation effects on blade characteristics began to become apparent. Further, the blade row model showed accurate simulation of the pressures in the interconnecting ducts by reproducing the wave and its harmonics up to approximately 118 Hz where the length of the volumes becomes too large to provide proper spatial definition of the harmonics of the standing waves although the fundamental frequency is properly reproduced.

This final report covers the research on low cost implementation of analysis and synthesis of speech using linear predictive coding (LPC). The report is in two completely separate parts which were originally the reports for Masters of Science projects by Paul Hurst (Part I) and Eugene Evancoe (Part II). Part I describes an approach to adaptive autocorrelation analysis which could be implemented using analog sampled data techniques for multiplication and filtering. Part II is an investigation of the use of high order switched-capacitor ladder filters for use as an LPC synthesis filter. (Author)

This report discusses four topics: (1) Operation of a frequency acquisition aid to acquire a direct sequence or frequency hopped signal. The acquisition aid employed is a FFT, and repeated measurements are employed to reduce the probability of error; (2) Multiple access using dynamic reservations. Here a new multiple access technique is described in which reservations are made which indicate whether the number of packets to be transmitted is to be increased or decreased. This technique is compared to the generally used procedure of indicating in the reservation the exact number of packets to be sent, a technique which requires significantly more bandwidth. (Author)

This report delineates the theory and terminology of data compression. It surveys four data compression methods--null suppression, pattern substitution, statistical encoding, and telemetry compression--and relates them to a standard statistical coding problem, i.e., the noiseless coding problem. The well defined solution to that problem can serve as a standard on which to base the effectiveness of data compression methods. The simple measure described for calculating the effectiveness of a data compression method is based on the characterization of the solution to the noiseless coding problem. Finally, guidelines are stated concerning the relevance of data compression to data processing applications. (Author/DAG)

A case study is presented where an image segmentation based compression technique is applied to LANDSAT Thematic Mapper (TM) and Nimbus-7 Coastal Zone Color Scanner (CZCS) data. The compression technique, called Spatially Constrained Clustering (SCC), can be regarded as an adaptive vector quantization approach. The SCC can be applied to either single or multiple spectral bands of image data. The segmented image resulting from SCC is encoded in small rectangular blocks, with the codebook varying from block to block. Lossless compression potential (LDP) of sample TM and CZCS images are evaluated. For the TM test image, the LCP is 2.79. For the CZCS test image the LCP is 1.89, even though when only a cloud-free section of the image is considered the LCP increases to 3.48. Examples of compressed images are shown at several compression ratios ranging from 4 to 15. In the case of TM data, the compressed data are classified using the Bayes' classifier. The results show an improvement in the similarity between the classification results and ground truth when compressed data are used, thus showing that compression is, in fact, a useful first step in the analysis.

An investigation is presented of video data compression applied to microgravity space experiments using High Resolution High Frame Rate Video Technology (HHVT). An extensive survey of methods of video data compression, described in the open literature, was conducted. The survey examines compression methods employing digital computing. The results of the survey are presented. They include a description of each method and assessment of image degradation and video data parameters. An assessment is made of present and near term future technology for implementation of video data compression in high speed imaging system. Results of the assessment are discussed and summarized. The results of a study of a baseline HHVT video system, and approaches for implementation of video data compression, are presented. Case studies of three microgravity experiments are presented and specific compression techniques and implementations are recommended.

Voice bandwidth compression techniques were investigated to anticipate link margin difficulties in the shuttle S-band communication system. It was felt that by reducing the data rate on each voice channel from the baseline 24 (or 32) Kbps to 8 Kbps, additional margin could be obtained. The feasibility of such an alternate voice transmission system was studied. Several factors of prime importance that were addressed are: (1) achieving high quality voice at 8 Kbps; (2) performance in the presence of the anticipated shuttle cabin environmental noise; (3) performance in the presence of the anticipated channel error statistics; and (4) minimal increase in size, weight, and power over the current baseline voice processor.

We present an off-line computational method for control allocation design. The control allocation function delta = F(z)tau = delta (sub 0) (z) mapping commanded body-frame torques to actuator commands is implicitly specified by trim condition delta (sub 0) (z) and by a robust pseudo-inverse problem double vertical line I - G(z) F(z) double vertical line less than epsilon (z) where G(z) is a system Jacobian evaluated at operating point z, z circumflex is an estimate of z, and epsilon (z) less than 1 is a specified error tolerance. The allocation function F(z) = sigma (sub i) psi (z) F (sub i) is computed using a heuristic technique for selecting wavelet basis functions psi and a constrained least-squares criterion for selecting the allocation matrices F (sub i). The method is applied to entry trajectory control allocation for a reusable launch vehicle (X-33).

A highly performing image data compression technique is currently being developed for space science applications under the requirement of high-speed and pushbroom scanning. The technique is also applicable to frame based imaging data. The algorithm combines a two-dimensional transform with a bitplane encoding; this results in an embedded bit string with exact desirable compression rate specified by the user. The compression scheme performs well on a suite of test images acquired from spacecraft instruments. It can also be applied to three-dimensional data cube resulting from hyper-spectral imaging instrument. Flight qualifiable hardware implementations are in development. The implementation is being designed to compress data in excess of 20 Msampledsec and support quantization from 2 to 16 bits. This paper presents the algorithm, its applications and status of development.

Data enhancement by data compression using polynomial fitting

The expected data rate produced by the Low Frequency Instrument (LFI) planned to fly on the ESA Planck mission in 2007, is over a factor 8 larger than the bandwidth allowed by the spacecraft transmission system to download the LFI data. We discuss the application of lossless compression to Planck/LFI data streams in order to reduce the overall data flow. We perform both theoretical analysis and experimental tests using realistically simulated data streams in order to fix the statistical properties of the signal and the maximal compression rate allowed by several lossless compression algorithms. We studied the influence of signal composition and of acquisition parameters on the compression rate Cr and develop a semiempirical formalism to account for it. The best performing compressor tested up to now is the arithmetic compression of order 1, designed for optimizing the compression of white noise like signals, which allows an overall compression rate = 2.65 +/- 0.02. We find that such result is not improved by other lossless compressors, being the signal almost white noise dominated. Lossless compression algorithms alone will not solve the bandwidth problem but needs to be combined with other techniques.

Optical wide-field surveys with a high cadence are expected to create a new field of astronomy, so-called "movie astronomy," in the near future. The amount of data of the observations will be huge, and hence efficient data compression will be indispensable. Here we propose a low-rank matrix approximation with sparse matrix decomposition as a promising solution to reduce the data size effectively, while preserving sufficient scientific information. We apply one of the methods to the movie data obtained with the prototype model of the Tomo-e Gozen mounted on the 1.0-m Schmidt telescope of Kiso Observatory. Once the full-scale observation of the Tomo-e Gozen commences, it will generate ~30 TB of data per night. We demonstrate that the data are compressed by a factor of about 10 in size without losing transient events like optical short transient point-sources and meteors. The intensity of point sources can be recovered from the compressed data. The processing runs sufficiently fast, compared with the expected data-acquisition rate in the actual observing runs.

A novel data compression scheme is presented. The method is very suitable for black and white images, and it can generate a compression factor of eight; in general the bitmap is optimized for an arbitary number of colors and not only for unused information contained in a conventional bitmap with $2^j$ bits per pixel. This compression method can be incorporated with other compression methods; it is suitable for other image types and audio. The potential high compression factor is cost effective for both memory and bandwidth requirements.

We define the complexity of DNA sequences as the information content per nucleotide, calculated by means of some Lempel-Ziv data compression algorithm. It is possible to use the statistics of the complexity values of the functional regions of different complete genomes to distinguish among genomes of different domains of life (Archaea, Bacteria and Eukarya). We shall focus on the distribution function of the complexity of noncoding regions. We show that the three domains may be plotted in separate regions within the two-dimensional space where the axes are the skewness coefficient and the curtosis coefficient of the aforementioned distribution. Preliminary results on 15 genomes are introduced.

We present a data compression and dimensionality reduction scheme for data fusion and aggregation applications to prevent data congestion and reduce energy consumption at network connecting points such as cluster heads and gateways. Our in-network approach can be easily tuned to analyze the data temporal or spatial correlation using an unsupervised neural network scheme, namely the autoencoders. In particular, our algorithm extracts intrinsic data features from previously collected historical samples to transform the raw data into a low dimensional representation. Moreover, the proposed framework provides an error bound guarantee mechanism. We evaluate the proposed solution using real-world data sets and compare it with traditional methods for temporal and spatial data compression. The experimental validation reveals that our approach outperforms several existing wireless sensor network's data compression methods in terms of compression efficiency and signal reconstruction.

This document is the proceedings from the 'Science Information Management and Data Compression Workshop,' which was held on October 26-27, 1995, at the NASA Goddard Space Flight Center, Greenbelt, Maryland. The Workshop explored promising computational approaches for handling the collection, ingestion, archival, and retrieval of large quantities of data in future Earth and space science missions. It consisted of fourteen presentations covering a range of information management and data compression approaches that are being or have been integrated into actual or prototypical Earth or space science data information systems, or that hold promise for such an application. The Workshop was organized by James C. Tilton and Robert F. Cromp of the NASA Goddard Space Flight Center.

We address the issue of applying quantized compressed sensing (CS) on low-energy telemonitoring. So far, few works studied this problem in applications where signals were only approximately sparse. We propose a two-stage data compressor based on quantized CS, where signals are compressed by compressed sensing and then the compressed measurements are quantized with only 2 bits per measurement. This compressor can greatly reduce the transmission bit-budget. To recover signals from underdetermined, quantized measurements, we develop a Bayesian De-quantization algorithm. It can exploit both the model of quantization errors and the correlated structure of physiological signals to improve the quality of recovery. The proposed data compressor and the recovery algorithm are validated on a dataset recorded on 12 subjects during fast running. Experiment results showed that an averaged 2.596 beat per minute (BPM) estimation error was achieved by jointly using compressed sensing with 50% compression ratio and a 2-bit quantizer. The results imply that we can effectively transmit n bits instead of n samples, which is a substantial improvement for low-energy wireless telemonitoring.

The First Geiger-mode Avalanche photodiode (G-APD) Cherenkov Telescope (FACT) has been operating on the Canary island of La Palma since October 2011. Operations were automated so that the system can be operated remotely. Manual interaction is required only when the observation schedule is modified due to weather conditions or in case of unexpected events such as a mechanical failure. Automatic operations enabled high data taking efficiency, which resulted in up to two terabytes of FITS files being recorded nightly and transferred from La Palma to the FACT archive at ISDC in Switzerland. Since long term storage of hundreds of terabytes of observations data is costly, data compression is mandatory. This paper discusses the design choices that were made to increase the compression ratio and speed of writing of the data with respect to existing compression algorithms. Following a more detailed motivation, the FACT compression algorithm along with the associated I/O layer is discussed. Eventually, the performances of the algorithm is compared to other approaches.

Realist, no-collapse interpretations of quantum mechanics, such as Everett's, face the probability problem: how to justify the norm-squared (Born) rule from the wavefunction alone. While any basis-independent measure can only be norm-squared (due to the Gleason-Busch Theorem) this fact conflicts with various popular, non-wavefunction-based phenomenological measures - such as observer, outcome or world counting - that are frequently demanded of Everettians. These alternatives conflict, however, with the wavefunction realism upon which Everett's approach rests, which seems to call for an objective, basis-independent measure based only on wavefunction amplitudes. The ability of quantum probabilities to destructively interfere with each other, however, makes it difficult to see how probabilities can be derived solely from amplitudes in an intuitively appealing way. I argue that the use of algorithmic probability can solve this problem, since the objective, single-case probability measure that wavefunction realism demands is exactly what algorithmic information theory was designed to provide. The result is an intuitive account of complex-valued amplitudes, as coefficients in an optimal lossy data compression, such that changes in algorithmic information content (entropy deltas) are associated with phenomenal transitions.

We propose geometric weighting as a novel method to combine multiple models in data compression. Our results reveal the rationale behind PAQ-weighting and generalize it to a non-binary alphabet. Based on a similar technique we present a new, generic linear mixture technique. All novel mixture techniques rely on given weight vectors. We consider the problem of finding optimal weights and show that the weight optimization leads to a strictly convex (and thus, good-natured) optimization problem. Finally, an experimental evaluation compares the two presented mixture techniques for a binary alphabet. The results indicate that geometric weighting is superior to linear weighting.

Motivated by the problem of the definition of a distance between two sequences of characters, we investigate the so-called learning process of typical sequential data compression schemes. We focus on the problem of how a compression algorithm optimizes its features at the interface between two different sequences $A$ and $B$ while zipping the sequence $A+B$ obtained by simply appending $B$ after $A$. We show the existence of a universal scaling function (the so-called learning function) which rules the way in which the compression algorithm learns a sequence $B$ after having compressed a sequence $A$. In particular it turns out that it exists a crossover length for the sequence $B$, which depends on the relative entropy between $A$ and $B$, below which the compression algorithm does not learn the sequence $B$ (measuring in this way the relative entropy between $A$ and $B$) and above which it starts learning $B$, i.e. optimizing the compression using the specific features of $B$. We check the scaling function on three main classes of systems: Bernoulli schemes, Markovian sequences and the symbolic dynamic generated by a non trivial chaotic system (the Lozi map). As a last application of the method we present the results of a recognition experiment, namely recognize which dynamical systems produced a given time sequence. We finally point out the potentiality of these results for segmentation purposes, i.e. the identification of homogeneous sub-sequences in heterogeneous sequences (with applications in various fields from genetic to time-series analysis).

The use of parity-check gates in information theory has proved to be very efficient. In particular, error correcting codes based on parity checks over low-density graphs show excellent performances. Another basic issue of information theory, namely data compression, can be addressed in a similar way by a kind of dual approach. The theoretical performance of such a Parity Source Coder can attain the optimal limit predicted by the general rate-distortion theory. However, in order to turn this approach into an efficient compression code (with fast encoding/decoding algorithms) one must depart from parity checks and use some general random gates. By taking advantage of analytical approaches from the statistical physics of disordered systems and SP-like message passing algorithms, we construct a compressor based on low-density non-linear gates with a very good theoretical and practical performance.

Thesis advisor,Murali Tummala

From the bitsavers.org collection, a scanned-in computer-related document. communications :: wellfleet :: router :: 110056A Customizing Data Compression Services 199502

We use the traveling front approach to derive exact asymptotic results for the statistics of the number of particles in a class of directed diffusion limited aggregation models on a Cayley tree. We point out that some aspects of these models are closely connected to two different problems in computer science, namely the digital search tree problem in data structures and the Lempel-Ziv algorithm for data compression. The statistics of the number of particles studied here is related to the statistics of height in digital search trees which, in turn, is related to the statistics of the length of the longest word formed by the Lempel-Ziv algorithm. Implications of our results to these computer science problems are pointed out.

We use the traveling front approach to derive exact asymptotic results for the statistics of the number of particles in a class of directed diffusion limited aggregation models on a Cayley tree. We point out that some aspects of these models are closely connected to two different problems in computer science, namely the digital search tree problem in data structures and the Lempel-Ziv algorithm for data compression. The statistics of the number of particles studied here is related to the statistics of height in digital search trees which, in turn, is related to the statistics of the length of the longest word formed by the Lempel-Ziv algorithm. Implications of our results to these computer science problems are pointed out.

Advances in audio data compression are largely driven by the need to conserve transmission rate or bandwidth, while maintaining the ability to accurately reconstruct the signal at the receiver. This report examines data compression methods with an emphasis on techniques for the compression of audio data. An overview of data compression schemes is presented to provide the background for a performance comparison between selected versions of data compression systems featuring adaptive differential pulse code modulation (ADPCM) schemes. Two different types of data compression systems are investigated; IIR and FIR impulse implementations. A modification to the basic ADPCM system using a modular function is implemented. The modular operation results in a smaller size codebook and prevents data expansion when the source is not matched to the code. This modification is utilized for both types of ADPCM coders compared. To complete the compression system Huffman coding, is employed to encode and decode the compressed data to and from binary form.

In radio interferometry, observed visibilities are intrinsically sampled at some interval in time and frequency. Modern interferometers are capable of producing data at very high time and frequency resolution; practical limits on storage and computation costs require that some form of data compression be imposed. The traditional form of compression is a simple averaging of the visibilities over coarser time and frequency bins. This has an undesired side effect: the resulting averaged visibilities "decorrelate", and do so differently depending on the baseline length and averaging interval. This translates into a non-trivial signature in the image domain known as "smearing", which manifests itself as an attenuation in amplitude towards off-centre sources. With the increasing fields of view and/or longer baselines employed in modern and future instruments, the trade-off between data rate and smearing becomes increasingly unfavourable. In this work we investigate alternative approaches to low-loss data compression. We show that averaging of the visibility data can be treated as a form of convolution by a boxcar-like window function, and that by employing alternative baseline-dependent window functions a more optimal interferometer smearing response may be induced. In particular, we show improved amplitude response over a chosen field of interest, and better attenuation of sources outside the field of interest. The main cost of this technique is a reduction in nominal sensitivity; we investigate the smearing vs. sensitivity trade-off, and show that in certain regimes a favourable compromise can be achieved. We show the application of this technique to simulated data from the Karl G. Jansky Very Large Array (VLA) and the European Very-long-baseline interferometry Network (EVN).

The quantum entropy-typical subspace theory is specified. It is shown that any mixed state with von Neumann entropy less than h can be preserved approximately by the entropy-typical subspace with entropy= h. This result implies an universal compression scheme for the case that the von Neumann entropy of the source does not exceed h.

In order to compress quantum messages without loss of information it is necessary to allow the length of the encoded messages to vary. We develop a general framework for variable-length quantum messages in close analogy to the classical case and show that lossless compression is only possible if the message to be compressed is known to the sender. The lossless compression of an ensemble of messages is bounded from below by its von-Neumann entropy. We show that it is possible to reduce the number of qbits passing through a quantum channel even below the von-Neumann entropy by adding a classical side-channel. We give an explicit communication protocol that realizes lossless and instantaneous quantum data compression and apply it to a simple example. This protocol can be used for both online quantum communication and storage of quantum data.

In order to promote public education and public safety, equal justice for all, a better informed citizenry, the rule of law, world trade and world peace, this legal document is hereby made available on a noncommercial basis, as it is the right of all humans to know and speak the laws that govern them. (For more information: 12 Tables of Code ) Name of Standards Organization: Bureau of Indian Standards (BIS) Division Name: Transport Engineering Section Name: Spring (TED 21) Designator of Legally Binding Document: IS 7906-3 Title of Legally Binding Document: Helical compression springs, Part 3: Data sheet for springs made from circular section wire and bar Number of Amendments: Equivalence: Superceding: Superceded by: LEGALLY BINDING DOCUMENT Step Out From the Old to the New --Jawaharlal Nehru Invent a new India using knowledge. --Satyanarayan Gangaram Pitroda

In todays digital world, big data is growing at an unprecedented rate, presenting significant challenges in terms of storage and management. Compression algorithms play a pivotal role in addressing these challenges by reducing data size without compromising essential information. This article provides an in depth exploration of data compression techniques, categorizing them into lossless and lossy compression methods, and evaluates their effectiveness in big data applications. The integration of these algorithms into popular big data frameworks such as Hadoop, Spark, and cloud storage systems is discussed, highlighting their impact on storage efficiency and data accessibility. A comparison of several compression algorithms, including Snappy, Zstandard, Gzip, and Brotli, is also presented to guide the selection of the most suitable algorithm based on compression ratio, speed, and the type of data being processed. The article concludes with a look into future trends and innovations in compression, including AI and machine learning based techniques, adaptive methods, and the potential for quantum compression. These advancements offer exciting prospects for improving data storage and processing capabilities, while addressing ongoing challenges in big data environments. Dr. Gopal Prasad Sharma | Prof. Raj Kumar Thakur | Prof. Dr. Pawan Kumar Jha "Compression Algorithms for Efficient Big Data Storage" Published in International Journal of Trend in Scientific Research and Development (ijtsrd), ISSN: 2456-6470, Volume-9 | Issue-3 , June 2025, URL: https://www.ijtsrd.com/papers/ijtsrd81127.pdf Paper URL: https://www.ijtsrd.com/computer-science/other/81127/compression-algorithms-for-efficient-big-data-storage/dr-gopal-prasad-sharma

In todays digital world, big data is growing at an unprecedented rate, presenting significant challenges in terms of storage and management. Compression algorithms play a pivotal role in addressing these challenges by reducing data size without compromising essential information. This article provides an in depth exploration of data compression techniques, categorizing them into lossless and lossy compression methods, and evaluates their effectiveness in big data applications. The integration of these algorithms into popular big data frameworks such as Hadoop, Spark, and cloud storage systems is discussed, highlighting their impact on storage efficiency and data accessibility. A comparison of several compression algorithms, including Snappy, Zstandard, Gzip, and Brotli, is also presented to guide the selection of the most suitable algorithm based on compression ratio, speed, and the type of data being processed. The article concludes with a look into future trends and innovations in compression, including AI and machine learning based techniques, adaptive methods, and the potential for quantum compression. These advancements offer exciting prospects for improving data storage and processing capabilities, while addressing ongoing challenges in big data environments. Dr. Gopal Prasad Sharma | Prof. Raj Kumar Thakur | Prof. Dr. Pawan Kumar Jha "Compression Algorithms for Efficient Big Data Storage" Published in International Journal of Trend in Scientific Research and Development (ijtsrd), ISSN: 2456-6470, Volume-9 | Issue-3 , June 2025, URL: https://www.ijtsrd.com/papers/ijtsrd81127.pdf Paper URL: https://www.ijtsrd.com/computer-science/other/81127/compression-algorithms-for-efficient-big-data-storage/dr-gopal-prasad-sharma

In todays digital world, big data is growing at an unprecedented rate, presenting significant challenges in terms of storage and management. Compression algorithms play a pivotal role in addressing these challenges by reducing data size without compromising essential information. This article provides an in depth exploration of data compression techniques, categorizing them into lossless and lossy compression methods, and evaluates their effectiveness in big data applications. The integration of these algorithms into popular big data frameworks such as Hadoop, Spark, and cloud storage systems is discussed, highlighting their impact on storage efficiency and data accessibility. A comparison of several compression algorithms, including Snappy, Zstandard, Gzip, and Brotli, is also presented to guide the selection of the most suitable algorithm based on compression ratio, speed, and the type of data being processed. The article concludes with a look into future trends and innovations in compression, including AI and machine learning based techniques, adaptive methods, and the potential for quantum compression. These advancements offer exciting prospects for improving data storage and processing capabilities, while addressing ongoing challenges in big data environments. Dr. Gopal Prasad Sharma | Prof. Raj Kumar Thakur | Prof. Dr. Pawan Kumar Jha "Compression Algorithms for Efficient Big Data Storage" Published in International Journal of Trend in Scientific Research and Development (ijtsrd), ISSN: 2456-6470, Volume-9 | Issue-3 , June 2025, URL: https://www.ijtsrd.com/papers/ijtsrd81127.pdf Paper URL: https://www.ijtsrd.com/computer-science/other/81127/compression-algorithms-for-efficient-big-data-storage/dr-gopal-prasad-sharma

Covariance and histogram image descriptors provide an effective way to capture information about images. Both excel when used in combination with special purpose distance metrics. For covariance descriptors these metrics measure the distance along the non-Euclidean Riemannian manifold of symmetric positive definite matrices. For histogram descriptors the Earth Mover's distance measures the optimal transport between two histograms. Although more precise, these distance metrics are very expensive to compute, making them impractical in many applications, even for data sets of only a few thousand examples. In this paper we present two methods to compress the size of covariance and histogram datasets with only marginal increases in test error for k-nearest neighbor classification. Specifically, we show that we can reduce data sets to 16% and in some cases as little as 2% of their original size, while approximately matching the test error of kNN classification on the full training set. In fact, because the compressed set is learned in a supervised fashion, it sometimes even outperforms the full data set, while requiring only a fraction of the space and drastically reducing test-time computation.

B. Schumacher and M. Westmoreland have established a quantum analog of a well-known classical information theory result on a role of relative entropy as a measure of non-optimality in (classical) data compression. In this paper, we provide an alternative, simple and constructive proof of this result by constructing quantum compression codes (schemes) from classical data compression codes. Moreover, as the quantum data compression/coding task can be effectively reduced to a (quasi-)classical one, we show that relevant results from classical information theory and data compression become applicable and therefore can be extended to the quantum domain.

Large data-sets defined on the sphere arise in many fields. In particular, recent and forthcoming observations of the anisotropies of the cosmic microwave background (CMB) made on the celestial sphere contain approximately three and fifty mega-pixels respectively. The compression of such data is therefore becoming increasingly important. We develop algorithms to compress data defined on the sphere. A Haar wavelet transform on the sphere is used as an energy compression stage to reduce the entropy of the data, followed by Huffman and run-length encoding stages. Lossless and lossy compression algorithms are developed. We evaluate compression performance on simulated CMB data, Earth topography data and environmental illumination maps used in computer graphics. The CMB data can be compressed to approximately 40% of its original size for essentially no loss to the cosmological information content of the data, and to approximately 20% if a small cosmological information loss is tolerated. For the topographic and illumination data compression ratios of approximately 40:1 can be achieved when a small degradation in quality is allowed. We make our SZIP program that implements these compression algorithms available publicly.

A study of data compression techniques involving linear interpolation and linear prediction showed that redundancy is a problem that can be significantly reduced by various polynomial approximations. A more recent compressor, the continuous secant compressor which determines the optimum sampling interval prior to sampling, was found to be the most efficient compressor examined. The continuous secant compressor bases its reduction technique on a straight-line approximation. Data compression results when the system in question does not occupy its entire bandwidth. The addition of white noise over the entire bandwidth was found to reduce the efficiency of the continuous secant compressor by only a small amount. The probability distribution of the straight-line approximation in the presence of noise had a gaussian distribution and a relatively small standard deviation.

Article from BYTE magazine, March 1991

We propose `Dracula', a new framework for unsupervised feature selection from sequential data such as text. Dracula learns a dictionary of $n$-grams that efficiently compresses a given corpus and recursively compresses its own dictionary; in effect, Dracula is a `deep' extension of Compressive Feature Learning. It requires solving a binary linear program that may be relaxed to a linear program. Both problems exhibit considerable structure, their solution paths are well behaved, and we identify parameters which control the depth and diversity of the dictionary. We also discuss how to derive features from the compressed documents and show that while certain unregularized linear models are invariant to the structure of the compressed dictionary, this structure may be used to regularize learning. Experiments are presented that demonstrate the efficacy of Dracula's features.

The work completed in the project dealt with the following areas of data compression and its applications: the design and analysis of sophisticated methods for prediction based on data compression techniques, with applications to prefetching, caching, and locality management. Fast, practical, and code-efficient implementations of arithmetic coding and other coding methods, for use in text and image compression. New methods for choosing motion vectors yielding substantially better rate-distortion tradeoffs for video compression in video-conferencing applications. Duke University recently filed a patent application for the work on prediction.

Performance of the NATO data compression algorithm in a noisy communication channel is superior to that of the present TDF data compression algorithm, provided that noise channel is not too harsh. Text is legible in a noise channel that has a bit error probability on the order 0.0003 (single bit errors). Multiple (burst) errors are not a problem provided the burst length does not exceed ten bits. Where actual burst length greatly exceeds the ten-bit (maximum) average expected in tactical situation noise, data compression algorithm performance may be poor, as with most data compression algorithms without a powerful noise protection algorithm. The NATO algorithm has enough compression to meet the current TDF performance specification, but its compression efficiency is lower than that of the TDF. The two algorithms have about the same complexity. (Author)

The feasibility of using a sequency-ordered Fast Walsh transform algorithm in a transformation-type ECG data compression scheme was demonstrated. The relationship of the mean square error between the original and reconstructed ECG signals as a function of the number of Walsh coefficients retained and the number of bits used to represent each coefficient was determined. A finite impulse response digital low-pass filter was tested as a means of improving the diagnostic characteristics of the reconstructed ECG waveforms. Suggestions for further work in this area are included. (Author)