Bash on Christian Hotz-Behofsitshttps://hotzbehofsits.com/tags/bash/Recent content in Bash on Christian Hotz-BehofsitsHugoen-usSat, 09 Jan 2021 13:54:18 +0100Merge large CSV files with headershttps://hotzbehofsits.com/post/merge_csv_with_header/Sat, 09 Jan 2021 13:54:18 +0100https://hotzbehofsits.com/post/merge_csv_with_header/<p>If you have ever worked with classic data warehousing tools, you may already know the problem: <em>The CSV export splits the output into different files, often with one file containing no more than 1 million observations.</em> However, for analysis purposes, this format is not always optimal and must be merged into a single file. In bash this is not a big problem with cat, but only if there is no header present. The BigQuery CSV table export, for example, adds a header to each CSV file and this must be taken into account when merging.</p>