How To Use JSON To Log A Basic File Copy Process

Create yourself a Console application project and let us dive right in and create our using directives of the different aspects of the system we will be working with. I should point out that this tutorial is meant to be improved and edited by you the reader to function as you want it to. Take it as the working example that it is. Code as follows:

using Newtonsoft.Json;
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Threading;

In our main method we have:

 internal static void Main(string[] args)
 {
 if (!Directory.Exists(Json_PathBuilder.JsonDir))
 Directory.CreateDirectory(Json_PathBuilder.JsonDir);
 if (!Directory.Exists(Json_PathBuilder.Destination_Path))
 Directory.CreateDirectory(Json_PathBuilder.Destination_Path);
 Thread ProcessFiles_Thread = new Thread(() => Process_Files(Environment.GetFolderPath(Environment.SpecialFolder.Desktop)));
 ProcessFiles_Thread.Name = "Directory_Searcher";
 ProcessFiles_Thread.Start();
 }

In the above source code, we are first checking if the directory exists for the class : Json_PathBuilder and the variable : JsonDir. If that directory doesn’t exist, we create it. We are essentially building up the data structure for our application to enable it to be able to operate properly without exception.

In the same block of code above, we then move to create a thread which will enumerate all of the files in a given directory. The directory I am using is the desktop folder. We then execute a new thread and pass the path to the thread to be be executed, and we also name the thread for convenience, just in case we may want to call the thread later and interact with it.

It is easier to find a thread in a thread collection if you know it’s unique ID or by name if you have named it. Once the thread executes, it will execute the Process_Files method which takes one parameter; that parameter is a string value for us to send the address we want to iterate all files and sub files from within that directory and any sub directories therein. The Process_Files method looks as follows:

 internal static void Process_Files(string in_Directory)
 {
 JSon_Builder.Prep_Json(Directory.GetFiles(in_Directory, "*.*", SearchOption.AllDirectories).ToList(), Json_PathBuilder.Destination_Path);
 }

Now that we are returning the files to a list, and passing that onto the JSon_Builder classes method Prep_Json, we also include the destination directory of where the files will be backed-up too when calling that method. Lets also give you a glimpse into the JSon_Builder class also:

 public static class JSon_Builder
 {
 private static int File_Incrementer { get; set; }

 public static void Prep_Json(List<string> listOf_Files, string file_Destination)
 {
 foreach (string file in listOf_Files)
 {
 if (File.Exists(file))
 {
 if (!File.Exists(Path.Combine(Json_PathBuilder.JsonDir, string.Concat(Path.GetFileName(Json_PathBuilder.Json_File).Substring(0, 7), $"_{File_Incrementer}.txt"))))
 File.Create(Path.Combine(Json_PathBuilder.JsonDir, string.Concat(Path.GetFileName(Json_PathBuilder.Json_File).Substring(0, 7), $"_{File_Incrementer}.txt")));
 if (File_ToLarge(Path.Combine(Json_PathBuilder.JsonDir, string.Concat(Path.GetFileName(Json_PathBuilder.Json_File).Substring(0, 7), $"_{File_Incrementer}.txt"))))
 {
 File_Incrementer++;
 string jSonFileName = string.Concat(Path.GetFileName(Json_PathBuilder.Json_File).Substring(0, 7), $"_{File_Incrementer}.txt");

 Write_Json(file, file_Destination, jSonFileName);
 }
 else
 {
 Write_Json(file, file_Destination, Path.Combine(Json_PathBuilder.JsonDir, string.Concat(Path.GetFileName(Json_PathBuilder.Json_File).Substring(0, 7), $"_{File_Incrementer}.txt")));
 }
 }
 }
 }

Lets start paying attention to line 5 of the above code. With the list of files we passed in from our directory searcher, we now want to check that those files actually still exist.

 foreach (string file in listOf_Files)
 {
 if (File.Exists(file))

Once we guarantee the file still exists, we begin to construct the path for the JSon file we will use to write out our log entries of this file and also perform a check to establish if the file to be copied already exists in the Archive folder where we will store the copied files.

if (!File.Exists(Path.Combine(Json_PathBuilder.JsonDir, string.Concat(Path.GetFileName(Json_PathBuilder.Json_File).Substring(0, 7), $"_{File_Incrementer}.txt"))))

So if the JSon log file doesn’t exist, we create it:

 File.Create(Path.Combine(Json_PathBuilder.JsonDir, string.Concat(Path.GetFileName(Json_PathBuilder.Json_File).Substring(0, 7), $"_{File_Incrementer}.txt")));

Once the file is created; you need to remember that this file may have been created already and while this file is being wrote too, we essentially want to make sure that we do not exceed that 3.5 to 4MB comfort zone. As that is generally considered the JSON file size limit. Of course you can exceed this file size but you risk most applications not being able to open and read it due to the excessive size of the file.

So what I done was; after each file write, I monitor the file size by calling the File_ToLarge method:

 if (File_ToLarge(Path.Combine(Json_PathBuilder.JsonDir, string.Concat(Path.GetFileName(Json_PathBuilder.Json_File).Substring(0, 7), $"_{File_Incrementer}.txt"))))

For anyone wondering why I didn’t use Path.Combine where I have used string.Concat instead. The reason for this, is because I wanted to combine the two strings, and Path.Combine would only insert unneeded slashes. Anyway…this way works better.

Once we establish the bool state of the method, this ultimately defines whether or not we enter this condition or the else condition, which essentially means that we need to begin writing to a new JSon file for logging to continue, because the old one is considered nearing to be at its max file size for a JSon file. Lets look at that else condition real quick:

 else
 {
 Write_Json(file, file_Destination, Path.Combine(Json_PathBuilder.JsonDir, string.Concat(Path.GetFileName(Json_PathBuilder.Json_File).Substring(0, 7), $"_{File_Incrementer}.txt")));
 }

Notice above how we increment the : File_Incrementer if the file is to large. You will notice this in the block of code above this one. But in this else condition, we simply write out the file if the file is not to big, and we don’t change the increment variable. With that, lets look at this File_ToLarge method:

 private static bool File_ToLarge(string current_jSonFile)
 {
 long fileSize = new FileInfo(current_jSonFile).Length;
 int maxSize = 3670016;
 if (fileSize > maxSize)
 return true;
 else
 return false;
 }

So given we don’t know the actual file size of every file, I’ve opted to use long as the variable to determine the file size using the FileInfo to ascertain the JSon files current file size by using and calling on the length property.

I’ve then set an integral maxSize which probably should have been made a constant value. But anyhow, we compare an analysis to ensure the files size is greater than the allowed limit, in which we return back to the method which called this function.

If the file has returned true, we begin changing the increment variable : File_Incrementer and construct a new file. Once the variable is incremented. See line 3 of the code below, and the actual code responsible for writing new JSon file names.

 if (File_ToLarge(Path.Combine(Json_PathBuilder.JsonDir, string.Concat(Path.GetFileName(Json_PathBuilder.Json_File).Substring(0, 7), $"_{File_Incrementer}.txt"))))
 {
 File_Incrementer++;
 string jSonFileName = string.Concat(Path.GetFileName(Json_PathBuilder.Json_File).Substring(0, 7), $"_{File_Incrementer}.txt");

 Write_Json(file, file_Destination, jSonFileName);
 }

Once we have the new filename built, we then pass it on to this method : Write_Json. This method takes the path of the origin file to be copied, along with the destination file path and the path to the JSon file which will be wrote too to record the data of the file copy process.

 private static void Write_Json(string file, string file_Destination, string toJson_File)
 {
 bool exists = File.Exists(Path.Combine(file_Destination, Path.GetFileName(file)));
 if (!exists)
 {
 Thread fileCopy_Thread = new Thread(() => FileCopy_Now(file, Path.Combine(file_Destination, Path.GetFileName(file))));
 fileCopy_Thread.Name = "FileCopy Process"; fileCopy_Thread.Start(); exists = true;
 }
 File.AppendAllText(Path.Combine(Json_PathBuilder.JsonDir, toJson_File), JsonConvert.SerializeObject(new DataStructure(file, Path.Combine(file_Destination, Path.GetFileName(file)), exists), Formatting.Indented));
 }

The Boolean exists variable checks if the destination file actually exists or not. If it does not exist, it is copied over to the Archive folder where it will be a backed up and stored by the file copy process.

That file copy process is commenced by way of executing a new thread namely the fileCopy_Thread and we execute this action by passing the original file path, with the Archive directory folder namely file_Destination and that is combined with the original file name that we acquired by calling the Path.GetFileName() method which returns only the name of the file and its extension and with this, we can construct the copied file name for the Archive directory by executing the thread once we have passed in these paths:

() => FileCopy_Now(file, Path.Combine(file_Destination, Path.GetFileName(file))));

And then started the thread:

fileCopy_Thread.Start(); exists = true;

 private static void FileCopy_Now(string origin, string destination) => File.Copy(origin, destination);

As you can see, we copy the file over above. But It is also worth noting that this is where you would essentially ensure that the destination file was successfully copied. Instead, I just set the exists boolean to true instead of checking that the file copy process actually succeeded instead of assuming that it did. This is where you will make your own logic for determining what the JSon file actually writes regarding this context.

 File.AppendAllText(Path.Combine(Json_PathBuilder.JsonDir, toJson_File), JsonConvert.SerializeObject(new DataStructure(file, Path.Combine(file_Destination, Path.GetFileName(file)), exists), Formatting.Indented));

Using File.AppendAllText; this will create the file if it doesn’t exist, and it it does exist, it will append the file contents and continue to write the new text at the bottom the already existing content. We combine the JSon directory with that of the JSon file that we passed to the Write_Json method earlier on. Using JsonConvert, we serialize the object of our DataStructure which looks like this:

 public class DataStructure
 {
 public DataStructure(string origin, string archive, bool exists_InArchive)
 {
 Origin = origin;
 Archive = archive;
 Exists_InArchive = exists_InArchive;
 }

 public string Origin { get; set; }
 public string Archive { get; set; }
 public bool Exists_InArchive { get; set; }
 }

By calling the constructor and returning the object variables we sent to it where we began writing the File.AppendAllText:

.SerializeObject(new DataStructure(file, Path.Combine(file_Destination, Path.GetFileName(file)), exists), Formatting.Indented));

The original file, and the file destination file (once combined), are sent along with the exists boolean to be written to the JSon log file. Finishing up by adding the overloads for the JsonConvert.SerializeObject() method which allows us to make our JSon file look pretty by adding Formatting.Indented.

Lastly, our Json_PathBuilder class is just a “holding place” for three variables for the directories structure that we have called numerous times above to construct on the fly:

 public static class Json_PathBuilder
 {
 public static readonly string JsonDir = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.Desktop), "Json Reports");
 public static readonly string Destination_Path = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.Desktop), "Backup Files");
 public static readonly string Json_File = Path.Combine(JsonDir, "JsonLog.txt");
 }

So after all of that, you should have a good understanding of what I’ve done and what you now need to do to properly ensure which files where actually successfully copied over to the Archive directory where the backups were stored. The complete source code for this project is as follows:

using Newtonsoft.Json;
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Runtime.InteropServices;
using System.Threading;

namespace TestConsoleApp
{
 internal static class Program
 {
 internal static void Main(string[] args)
 {
 if (!Directory.Exists(Json_PathBuilder.JsonDir))
 Directory.CreateDirectory(Json_PathBuilder.JsonDir);
 if (!Directory.Exists(Json_PathBuilder.Destination_Path))
 Directory.CreateDirectory(Json_PathBuilder.Destination_Path);
 Thread ProcessFiles_Thread = new Thread(() => Process_Files(Environment.GetFolderPath(Environment.SpecialFolder.Desktop)));
 ProcessFiles_Thread.Name = "Directory_Searcher";
 ProcessFiles_Thread.Start();
 }

 internal static void Process_Files(string in_Directory)
 {
 JSon_Builder.Prep_Json(Directory.GetFiles(in_Directory, "*.*", SearchOption.AllDirectories).ToList(), Json_PathBuilder.Destination_Path);
 }
 }

 public static class JSon_Builder
 {
 private static int File_Incrementer { get; set; }

 public static void Prep_Json(List<string> listOf_Files, string file_Destination)
 {
 foreach (string file in listOf_Files)
 {
 if (File.Exists(file))
 {
 if (!File.Exists(Path.Combine(Json_PathBuilder.JsonDir, string.Concat(Path.GetFileName(Json_PathBuilder.Json_File).Substring(0, 7), $"_{File_Incrementer}.txt"))))
 File.Create(Path.Combine(Json_PathBuilder.JsonDir, string.Concat(Path.GetFileName(Json_PathBuilder.Json_File).Substring(0, 7), $"_{File_Incrementer}.txt")));
 if (File_ToLarge(Path.Combine(Json_PathBuilder.JsonDir, string.Concat(Path.GetFileName(Json_PathBuilder.Json_File).Substring(0, 7), $"_{File_Incrementer}.txt"))))
 {
 File_Incrementer++;
 string jSonFileName = string.Concat(Path.GetFileName(Json_PathBuilder.Json_File).Substring(0, 7), $"_{File_Incrementer}.txt");

 Write_Json(file, file_Destination, jSonFileName);
 }
 else
 {
 Write_Json(file, file_Destination, Path.Combine(Json_PathBuilder.JsonDir, string.Concat(Path.GetFileName(Json_PathBuilder.Json_File).Substring(0, 7), $"_{File_Incrementer}.txt")));
 }
 }
 }
 }

 private static bool File_ToLarge(string current_jSonFile)
 {
 long fileSize = new FileInfo(current_jSonFile).Length;
 int maxSize = 3670016;
 if (fileSize > maxSize)
 return true;
 else
 return false;
 }

 private static void Write_Json(string file, string file_Destination, string toJson_File)
 {
 new JsonSerializer().Formatting = Formatting.Indented;
 bool exists = File.Exists(Path.Combine(file_Destination, Path.GetFileName(file)));
 if (!exists)
 {
 Thread fileCopy_Thread = new Thread(() => FileCopy_Now(file, Path.Combine(file_Destination, Path.GetFileName(file))));
 fileCopy_Thread.Name = "FileCopy Process"; fileCopy_Thread.Start(); exists = true;
 }
 File.AppendAllText(Path.Combine(Json_PathBuilder.JsonDir, toJson_File), JsonConvert.SerializeObject(new DataStructure(file, Path.Combine(file_Destination, Path.GetFileName(file)), exists), Formatting.Indented));
 }

 private static void FileCopy_Now(string origin, string destination) => File.Copy(origin, destination);
 }

 public class DataStructure
 {
 public DataStructure(string origin, string archive, bool exists_InArchive)
 {
 Origin = origin;
 Archive = archive;
 Exists_InArchive = exists_InArchive;
 }

 public string Origin { get; set; }
 public string Archive { get; set; }
 public bool Exists_InArchive { get; set; }
 }

 public static class Json_PathBuilder
 {
 public static readonly string JsonDir = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.Desktop), "Json Reports");
 public static readonly string Destination_Path = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.Desktop), "Backup Files");
 public static readonly string Json_File = Path.Combine(JsonDir, "JsonLog.txt");
 }
}