Code Asset inventory Overview

This article is an introduction to my SVN server.  My SVN server contains 22 years of custom code archives that attempt to reinvent just about every wheel ever invented.  I have always invented my own architectures when possible in order to 1) Fully understand various types of programming tasks and 2) Create interfaces that are programmatically more convenient and/or superior to what I find from 3rd parties.    This article will just scratch the surface of what’s available and where it is, but will not dive into the specifics of how each is constructed.  All of these frameworks may not be superior to other frameworks that exist out there, however, they served as a jumping off point that allowed me to create things that were more tailored to the purposes I had in mind.

  1. BigBrainUltra.pas A custom memory manager.    Years ago I created a custom heap manager that was multi-thread friendly.   I have some strange fascination with threads, and have built multi-CPU systems since I could afford to buy the parts.   “Big Brain” was once upon a time released as a commercial product, and it allows memory operations to scale across threads.  It essentially acts as a memory cache that is local to each thread in the program.  In order to use it, put BigBrainUltra.pas and BrainWashUltra.pas at the beginning of your project.  It won’t work if it isn’t the first used unit in your project file.  Then all of your apps memory operations, including those used by strings will be handled by BigBrain and will scale across threads, cores, and CPUSs.  It also has a DLL format, BigBrainShareMem.dll that enables it to be used with BPLs.
  2. SimpleAbstractConnection.pas  contains TSimpleAbstractConnection, which is the basis for communication pipes of all flavors.  It attempts to put just about any kind of I/O pipe under a single interface.  From it I derive various I/O pipes.  TSimpleWinsockConnection, TSimpleCommPort, TSimpleBLEConnection, TSimpleNamedPipeConnection, TSimpleReliableUDPConnection… etc… etc..
  3. SimpleReliableUDP.pas  contains what is essentially a TCP replacement built from UDP.   Once upon a time I grew tired of dealing with the limitations of and issues with TCP and replaced it with my own thing.  This was a much bigger challenge than I imagined, but after a couple of years of fussing around, I have something that is a pretty good replacement with a just a few unfortunate bugs.  This is a system that I would really love to get peer reviewed.
  4. CommandProcessor.pas contains TCommand and TCommandProcessor most notably.   Derive your own commands from TCommand and override DOExecute().  You may also want to override InitExpense() to set limits on resource usage.  To use your command, create it, Set your custom properties (if you define any), and then call Start().   The command will run in the background using an advanced thread-pool (see #5).  You can check IsComplete on your command to see if it is done, or call WaitFor().   Free it after extracting your results from it.  It is possible to use properties such as CPUExpense, Network Expense, and MemoryExpense, or the more-advanced Resources, property to control the flow of commands.  TCommandProcessor will attempt to run the commands in an optimal order based on available resources and the resource intensiveness of each command.

    For example, if you have 8 cores and run commands that have CPUExpense set to 0.5, the TCommandProcessor will run up to 16 of those commands at a time.  More advanced resource tagging is also possible, against custom resources that you define.  The idea is to allow TCommandProcessor to complete all the commands given to it in the most efficient way possible.  It is even possible to define command dependencies so that certain commands are not started until the commands they depend on are complete.

  5. ManagedThread.pas contains TThreadManager and TManagedThread.  TManagedThread is a wrapper around TThread that aims to unify the mess and disagreement that ultimately surrounds how programmers deal with threads.   Futhermore, Delphi’s TThread has literally 1,000,000 holes in it, and the use-cases for it I’ve seen on the web are way worse. If you seek advice on the web regarding the best way to handle threads in your app, you’ll inevitably get 50 responses that bear no resemblance to each other.  TManagedThread is my attempt at creating “one thread to rule them all” and is based on an analysis of every kind of thread use case that I see potentially existing.     It covers 1) Fire-once threads 2) Fire-Forget threads (without truly abandoning the thread pointers) 3) Thread-pooling 4) Active Looping threads 5) Passive looping threads 6) Separate hot/cold run intervals 7) Uniform termination methodology
  6. Packet.pas, RDTPProcessor.pas — RDTP  or “Relational Data Transfer Protocol” and related servers, can be thought of as a means to send data to and from a server in a binary format that is easy to use.  I’m not a fan of sending XML or JSON through my transport layers.  Instead I favor using strong types.  TRDTPPacket facilitates creating and interpreting binary messages that go to and from a server.   It even supports data compression and some basic encrpytion.  I use RDTP for all my custom clients and servers which are all built around a code generator, which is also in the SVN called RDTPGen.exe.  RDTPGen basically allows you to quickly create clients AND servers without messing around with data marshalling issues.  With just a few keystrokes and a little know-how, you can make calls to your back-end look essentially just like any other function call.  As an example, take a look at the code RDTPBeeperServer.pas.  As a simple test, I built a server that allows a client to call a single function, Beep(), to make a server running the back end Beep!  I use RDTP for lots of things, including my SQL middleware.RDTP calls are defined essentially in comments at the top of a unit and in text files under the RDTP_DEFINITIONS folder.  For example, here’s a file that represents that calls that I can make to my database server.

    When the code generator runs, it sets up scaffolding to marshal and unmarshal types in both Client and Server classes.   Literally literally all you have to do after running the generator is put a few lines of code into the back-end specific to your task.  Typically you don’t even have to touch the clients at all, with the only exception being if you want to implement a callback.  Even better is the fact that the code generator can be run more than once.  So if you add new functions, or change the signatures of the functions, new code is generated, while your custom code is preserved through inheritance.

  7. BetterObject.pas contains TBetterObject which is generally the basis for all objects I create.  It doesn’t add very much other than  Init(), Detach() functions, and a couple pointers for cases when you need to build linked lists without external linkage objects.  Detach() is particularly important to me because it is my way of dealing with the tremendous differences surrounding reference counting on all the Delphi platforms.
  8. SharedObject.pas is a basis for most objects that are shared across threads.  In a nutshell it is mostly just a TBetterObject with added Lock and Unlock functions.  However, it is worth noting that there are some alternatives to TSharedObject available, such as TLockQueuedObject which supports separate Read and Write locks and enforces a “wait your turn” policy to prevent threads from hogging resources.   Also there’s TNamedLocks which allows you to define locks that have names.  This is useful for locking all sorts of things… like a named path or URL (to make sure two threads are calling the same URL at the same time).  I actually use it in caches, for example, when I want to download URLs from the web and cache them on disk and want to make sure that only one thread is fetching a given URL at the same time.
  9. JSONHelpers.pas  contains a custom JSON parser.  I was not happy with the JSON parsers available to me by default, so I built my own.  My parser is a bit easier to use for a few reasons.  1) It supports array properties indexed by variants, so you can easily lookup subnodes for example jsonobject[‘results’][‘SomeArrayOfData’][45][‘Bid’].Value uses both named and indexed nodes in an easy-to-read manner.  My JSON parser is very new and still somewhat of an embarassing mess.  Be warned.  It is very much in active developement though.
  10. FlexiMath.pas is a very new pet project which aims to be a number-cruncher’s heaven, but I am just beginning to work on this.  The vision behind it is similar to that of a spreadsheet.   However, spreadsheets are very 2-dimensional entities.  FlexiMath is intended to implement just about anything you might do in a spread sheet, but from JSON documents.  So, for example, with just a couple of commands you could extract a bunch of values into an array, do a standard deviation against them, then store the results in another node in the JSON document that you then feed to other equations.  By keeping all the math in JSON documents, the documents can then be easily saved and loaded for the purpose of machine-learning, reporting, forensic analysis, archiving…. etc.   This system is going to be the core of all projects I do that involve heavy amounts of number crunching going forward.
  11. Advanced Streams.  Checkout TMultiBufferedMemoryFileStream, which provides some nice multi-threaded file buffering.  TUnbufferedFileStream for working with unbuffered files, TQueuedFileStream for working with file streams in with write-behind queue operations and side-buffer fetches.

Other odds and ends that are super interesting

  1. iSCSI server.  I wrote an iSCSI server that is wicked awesome.  It supports heterogeneous RAID (mixing RAID 0,1,5 on the same disk array), Real-time backups (as opposed to periodic backups), hot-swapping, and multi-tiered storage (keeps your most-used data on SSDs).  The core of it can be found in iSCSI.pas and VirtualDisk_Advanced.pas
  2. Sound Engine.   There is a sound engine that basically works both on Android and Windows that can be used to synthesize sounds that even features an audio effects stack with Fourier transform capabilities.  I used it to create a DJ audio system that interacts with DMX lighting.  Checkout SoundTools.pas, SoundInterfaces.pas and related files.
  3. DMXObject.pas contains components for working with DMX Lighting.  I have a passion for intelligent concert lighting and have outfitted my house with various lasers and arena-rock lights.
  4. 3D-Accelerated 2D engine.  See DX2D.pas.  It is kinda rudamentary, but I use this one a lot.
  5. 3D-Engine.   I heavily customized the PXL Engine, a 2D-Game Engine and turned it into a slightly more-proper 3D engine with better Shaders and proper gamecomponents inspired by Unity.