Deduplication Enabled Operating system Solution
Hardware required:
1) Server class Grid:
Cluster enabled grid of computers with no single point of failure to disks, network connectivity, memory modules, power supplies, RAIN architecture enabled.
2) Server Class Machine for base machine.
New Software Required:
1) Cluster enabled Microsoft server OS which has good cache manageability.
2) Software Services to maintain data integrity across grid.
3) Microsoft custom built de duplication plug-in which takes care of data integrity.
4) A cluster plug-in is required which manages data and resources across the nodes.
5) Management applications for reporting, managing through web/RDP etc.
Process Flow:
1) OS is installed on all the nodes of grid.
2) Base node is configured for management and hosting applications.
3) Base Unique blocks are placed on the data nodes. These are custom blocks which are derived after some research on various combinations of applications.
4) Grid is placed into production.
5) Read and write requests from client machines over Ethernet arrive to the server grid.
6) The enterprise de-dupe OS with its effective cache management system, decides whether a similar data/file block is available in the file system.
7) If the data/file block is not available in the base block, it is written to the disk, other wise a pointer is made into the disk.
8) The abstraction layer of the Global de duplication cache helps the multiple applications to share the storage, network, memory and CPU.
Advantages:
1) De-duplication is achieved at file system level, maximizing disk usage, of course with expense of RAM.
2) CIO/CTO looks at the machine as a single entity, so maintenance free.
3) Because Data blocks are designed in advance and placed readily into the data nodes, disk I/O is lesser.
4) Multiple applications can be hosted on to the same grid, DE-DUPE enabled.
5) The “server” concept takes a different shape.
6) High throughput as most of the activity is managed by Cache.
7) Ready made blocks reduces amount of Disk I/O.
8) Forms a great solution with good hardware and software applications.
9) Partnership with hardware vendors gets enriched.
10) Its advantageous to have an integrated disk based Backup and Restore to pull and push data to and from data nodes.
What we have to start off:
1) NTFS file system is owned by Microsoft. Applications like SQL, ERP, Web application, Share point are designed and developed by Microsoft.
2) Some reverse engineering + research and analysis on data /file blocks being produced with this type of applications can be easily found out and the same can be placed onto the file system as base blocks for duplicate blocks.
