Merge CSV Files in Specified Folder Using Java?

[probey20]

I'm not sure if I'm posting in the correct category, but I am trying to figure out how to merge several CSV files into one new file without having to specify all the file names. I just want to point to the correct folder and either run a Java program, batch file, or possibly an XSL transformation which merges the files.

Is what I'm trying to accomplish possible with these languages? I know it is possible in VB but I do not have VB installed at work or much experience programming in it.

Thanks!

[Aradon]

Yes, it is possible in Java. I don't know about the rest. Since a csv file is just comma's, tabs, and line returns, it would be easy to parse through the files, and combine them as needed.

Let us know if you need any help coding it (we wont do it for you but we can point you in the right direction). Also if you are new to java, I would suggest reading our FAQ in order to get familier with some tutorials and tools.

[probey20]

I'm no Java expert by any means, but I have coded programs that open a database connection, parse through the records and create XML files. I'm hoping this will be a similar process.

If you could point me in the general direction of how to access and parse CSV files in a specified folder, that would be great!

Thanks!

[brad211987]

If you are using java 5 or later, its very easy to use a Scanner, class is at java.util.Scanner. You can give it a reference to a File object and read it line by line.

API link: http://java.sun.com/j2se/1.5.0/docs/api/index.html?index-filesindex-1.html

*EDIT*
Link is partially broken, doesn't go directly to Scanner, it's in the class list in the left frame though.

[ess]

Just created a simple class that would read as many CVS files in a given directory and merge them into one. The contents of the CVS files are not checked in this class...I left that for you to do. This is just an example to demonstrate how to merge multiple files into one file...


import java.io.*;

public class Appender {

private BufferedReader console; // used to prompt users for input
private File directory; // directory containing cvs files
private File[] files; // all cvs files in a given directory
private final String fileName = "appender_app.cvs"; // name of the file containing final output

public Appender() {
// initialize console to start prompting user for input
initConsole();
// prompt user to enter directory containing files
System.out.print("Please Enter Directory [empty for current directory]: ");
String dir = this.getInput();
// check user input
if ("".equals(dir)) {
// if user didn't enter anything...use current directory
this.directory = new File(System.getProperty("user.dir"));
} //-- end if block
else {
// use user input as directory
this.directory = new File(dir);
} //-- ends else block

// check that user's input points to a directory that exists
// in the file system
if (this.directory.exists() && this.directory.isDirectory()) {
// check that the directory is both readable and writable
if (!this.directory.canRead() && !this.directory.canWrite()){
System.out.println("\n++++++++++++++++++++++++++++++++++++++++++++++++++");
System.out.println("Error:\nApplication will now terminate because:");
System.out.println("Directory: " + this.directory.getAbsolutePath());
System.out.println("Either; NOT WRITABLE or NOT READABLE");
System.out.println("++++++++++++++++++++++++++++++++++++++++++++++++++\n");
System.exit(-100);
} //-- end if block
// read directory and check for any cvs files
this.readSelectedDirectory();
// read all cvs files and append data to fileName
this.readAndAppend();
// close console once done
this.closeConsole();
} //-- end if block
else {
System.out.println("\n++++++++++++++++++++++++++++++++++++++++++++++++++");
System.out.println("Error:\nApplication will now terminate because:");
System.out.println("Directory: " + this.directory.getAbsolutePath());
System.out.println("Does not exist!!!!");
System.out.println("++++++++++++++++++++++++++++++++++++++++++++++++++\n");
System.exit(-100);
} //-- ends else block
} //-- ends class constructor

public void readSelectedDirectory() {
// inform the user of the working directory
System.out.println("Now Working in: " + this.directory.getAbsolutePath());
// get a list of all cvs files in a given directory
// Note: use of CVSFilter declared below this class
this.files = this.directory.listFiles(new CVSFilter());
// if a list of cvs files exist, continue...otherwise terminate application
if (files.length<2){
System.out.println("\n++++++++++++++++++++++++++++++++++++++++++++++++++");
System.out.println("Application will now terminate as the number of ");
System.out.println("files to read is zero or below 2 files");
System.out.println("++++++++++++++++++++++++++++++++++++++++++++++++++\n");
} //-- end if block

} //-- ends readSelectedDirectory

private void readAndAppend() {
File fileOut = new File(this.directory, fileName);
// if file exists...delete it...so it is ready for a fresh input
if (fileOut.exists()){
fileOut.delete();
} //-- end if block
// now create it
try {
fileOut.createNewFile();
System.out.println("File: '" + fileOut.getAbsolutePath() + "' created successfully ;)" );
} //-- ends try block
catch (IOException err ) {
err.printStackTrace(System.err);
} //-- ends catch block

// initialize buffer and start reading data
BufferedWriter out = null;
try {
out = new BufferedWriter(new FileWriter(fileOut,true));
for(File input : this.files) {
if (fileName.equalsIgnoreCase(input.getName())){
continue;
} //-- end if block
BufferedReader in = null;
try {
in = new BufferedReader(new FileReader(input));
System.out.println("++ Now reading: '" + input.getName() + "'");
String txt = null;
while ((txt=in.readLine())!=null) {
//System.out.println(in.readLine());
out.write(txt);
out.newLine();
out.flush();
} //-- ends while loop
in.close();
System.out.println("-- Finished Writing: '" + input.getName() + "'");
} //-- ends try
catch (IOException e ) {
e.printStackTrace(System.err);
} //-- ends catch block
} //-- ends for loop
} //-- ends try block
catch (FileNotFoundException err ) {
err.printStackTrace(System.err);
} //-- ends catch block
catch (IOException err ) {
err.printStackTrace(System.err);
} //-- ends catch block
finally {
try {
if(out!=null) {
out.close();
} //-- ends if block
} //-- ends try block
catch (IOException ignored ) {
//ignored.printStackTrace(System.err);
} //-- ends catch block
} //-- ends finally block

} //-- ends readAndAppend

private void initConsole() {
if (this.console==null){
this.console = new BufferedReader(new InputStreamReader(System.in));
} //-- end if block
} //-- ends initConsole

private void closeConsole() {
if (this.console!=null){
try {
this.console.close();
} //-- ends try block
catch (IOException ignored ) {
// ignored exception....but you can uncomment the line below
// ignored.printStackTrace(System.err);
} //-- ends catch block
} //-- end if block
} //-- ends close

private String getInput() {
String input = new String("");
try {
input = this.console.readLine();
} //-- ends try block
catch (IOException e ) {
e.printStackTrace(System.err);
} //-- ends catch block
return input;
} //-- ends getInput

public static void main(String[] args) {
new Appender();
} //-- ends class method main

} //-- ends class definition

// this class allows only listings of cvs files
// when reading directory etc.
class CVSFilter implements FilenameFilter {
public boolean accept(File dir, String name) {
return (name.endsWith(".cvs"));
} //-- ends accept
} //-- ends class definition


Let's know if it worked ok for you.
Cheers
~E

[probey20]

I will give this a try. Thank you very much!!

[praveenoruganti]

sarah,

Try merging all the files which is having any multiple folders i.e... you need to loop the folder and afterwards you need to merge it...

Try this if you are not able to proceed i will help you...

[shyam]

my 2 cents...write up a batch file/shell script if you are simply appending several csvs to create a single csv...what requires 100 odd lines in java just requires 4 lines in windows batch file (and even less in *nix)

echo off
for %%a in (*.csv) do (
type %%a >> all.csv
)

find . -type f -name '*.csv' -exec cat '{}' >> all.csv \;

[servlet]

I have the similar requirement, so I am writing a Java utility for it. will post it here, once it is complete.

---------------
SN
Java Servlet central (http://www.jsptube.com)

[servlet]

I wrote following utility which can merge any type of text files.

Note: It will not work with binary files.

public class FileMergeUtil {

public void merge(String[] sourceFiles, String destinationFile) throws IOException {
FileReader in;
ConcatReader concatReader = new ConcatReader();

for(int i=0; i< sourceFiles.length; i++) {
in = new FileReader(sourceFiles[i]);
concatReader.addReader(in);
}

concatReader.lastReaderAdded();

FileWriter fw = new FileWriter(destinationFile);

int c;
while ((c = concatReader.read()) != -1) {
fw.write((char)c);
}
fw.close();
concatReader.close();
}


Usage
- Pass the String array of source file paths and string destination file path as argument to merge method

Example
public static void main(String args[]) {
try {
new FileMergeUtil().merge(new String[] {
"src/csv/file1.txt",
"src/csv/file2.txt",
"src/csv/file3.txt" },
"src/csv/temp.csv");
} catch (IOException e) {
e.printStackTrace();
}
}



-- Dependency
It depends on OsterMillers ConcatReaderutility (http://ostermiller.org/utils/Concat.html).
So download and put the OsterMiller utility jar (http://ostermiller.org/utils/download.html) into classpath.

Java servlets (http://www.jsptube.com)