By Michelle Scardino and Mark O'Connell, EMC Atmos CTO
The other night I was shopping online for furniture. I recently completed a construction project and was excited to browse my favorite store for design and layout ideas that best fit my new space. Unfortunatley when I visited the site, it returned:
“We are temporarily updating our site, please come back soon.”
So I tried again in fifteen minutes, then in an hour, and one final time two hours later. Still unavailable. While frustrated, it sparked the idea to write this blog and discuss how EMC Atmos, with its policy-based active /active architecture may have enabled this company to non-disruptively update their system, keep me on their site, and quite possibly prevent me from buying a couch from their biggest competitor.
It all Starts with an Active/Active Architecture
Atmos' scale-out object design enables an active/active architecture - a critical enabler for a true cloud storage platform. Both reads and writes can be issued against any node, in any site, within the Atmos cloud, and Atmos software automatically detects where data is located and will access the optimal copy of the data, given the locality of the I/O request and the location(s) of the object within the system.
This same intelligence is applied when hardware components of the system fail, where the Atmos cloud can restore each object’s SLA after a failure as long as there is a readable copy of the object somewhere in the system. These mechanisms eliminate any manual intervention to bring disaster recovery copies online and thus provide a true high availability data solution at an unprecedented scale.
Many customers ask us “how” an active/active architecture actually works or if it can "really" eliminate the need for dedicated replication or back up. The answers to these questions are found in key features Atmos such as Replication and GeoParity as well as recommended configurations.
Atmos Replication: Synchronous and Asynchronous
Atmos supports two forms of replication: synchronous replication and asynchronous replication. For a particular object, both types of replication can be specified, depending on the needs of the application and the criticality of the data.
Replica definition for an object class is configured at the policy specification screen by the tenant administrator (We’ll talk more about this in the next section). It is the first step in determining where copies of an object will reside. These replicas, or copies, can be used to drive availability, disaster recover, content distribution and read performance. Replicas ensure that data is available despite hardware failures, that users, regardless of location, may have a copy of the data close to them to read from. It is important to determine how many, which location(s) and which type of replicas need to be defined for each class of object.
Synchronous replicas are those which the Atmos client services writes before returning an acknowledgement of successful write to the requesting application. The client services perform all synchronous writes, to all specified destinations, in parallel. Synchronous replication is typically used for applications which have very strict state requirements and which require a fully consistent and up-to-date data set in the case of a site outage, while the disadvantage is that the application ingest speed is typically bottlenecked by the speed of the synchronous replication, especially if it must traverse a WAN to a remote location.
Asynchronous replicas are used in policy with one or more synchronous replicas, and are usually placed in a separate location (RMG). The big advantage of asynchronous replicas is that there is no latency penalty for the application, which is especially important for remote copies which may traverse a higher latency WAN. Asynchronous replicas may also be used to optimize read performance for remote sites, as a read initiated at the remote site will use a local asynchronous copy if it exists and will traverse the WAN for the data only if the asynchronous replica has not yet been created, thereby generally providing extremely low latency reads at minimal cost to the application.
The number, location, and type of replica for each individual object can be controlled by utilizing metadata tags associated with the object. Through policy, each replica can then be assigned a specific location (E.g. – “sameAs” or “otherThan” “Boston”), or can utilize the “$client” tag, which signifies the Atmos node where the write comes into. In this way the data can be classified at a fine grained level and the appropriate protection applied, rather than applying a blanket protection policy for all data which overprotects less critical data and underprotects more critical data.
GeoParity is a policy-based replication option that offers excellent data durability characteristics while minimizing protection overhead. Atmos offers two options, one option which protects against six concurrent drive failures with 60% protection overhead, and a second option which protects against three concurrent drive failures with 33% overhead. Conceptually GeoParity is similar to RAID algorithms in that the raw data is split into multiple fragments and an additional redundancy fragment is calculated which can be used to recreate any of the original raw fragments if one of these should fail. The difference is that GeoParity uses a more complex encoding scheme for the redundancy fragments such that the data can be recreated even if multiple of the original fragments are lost. Atmos places the GeoParity fragments in independent fault domains, residing on different disks, on different nodes, and potentially even different RMGs within a system. This overall process is referred to as “encoding” the object. When an object is then requested by a client, Atmos will then “decode” the fragments back into the original object.
The Nuts and Bolts behind GeoParity
Atmos implements GeoParity with the Cauchy Reed-Solomon algorithm, and uses two different implementations. The first is a 9/12 configuration, where an object is split into 9 data fragments and 3 coding fragments. This effectively tolerates up to 3 drive failures, and gives the object 5 nines of data availability. In addition, the storage overhead is only 33%, compared with 100% for a RAID-1 configuration, or 25% for a RAID-5 configuration (using k/m to calculate overhead: 3/9, 1/1 or ¼). Such benefits come at a price, in this case some performance overhead is required for the encoding and decoding operations.
The second configuration is a 10/16 option, where there are 10 data fragments and 6 coding fragments. This configuration will tolerate up to 6 drive failures, has a 60% storage overhead, but at a cost of additional performance overhead than the 9/12 configuration.
There is an inherent MD5 inline integrity checksum that is calculated with each encoding operation on the fragments for erasure coding. Once a client requests the object, the checksum is calculated again against the stored checksum. If the checksum does not match for any fragments, that fragment or fragments is dropped.
Recommended Configurations for an Active/Active Architecture
If my furniture company had been using Atmos that fateful night, they would have avoided the problems that ultimately drove me to another site. For instance, consider the system below. Data is stored using a 9/12 GeoParity scheme for excellent data durability, and based on my locality, when I browse for furniture, I am directed to the nearest copy for fast response time as I browse options.
However, what happens if the local site is down? For example, for a maintenance activity (e.g. repairing AC at the site, upgrading power drops).
With an Active/Active architecture and GeoParity instead of returning “site temporarily unavailable”, the Atmos system would have recognized that the site was down and transparently redirect me to a remote site where I could have continued to browse through furniture options. What’s more, all of this is done automatically with no change to the application, no action by the administrator, and no impact to my browsing habits! There may have been a slightly higher latency, as my requests can no longer access the most optimal data copy, but any increase in latency would be a minor (a few seconds or so).
So in the end, I purchased an incredible couch (much like the one the yellow lab is on above), had a great online and in store experience, and found a great reason (they get technology) to love the competition.
Check out These Resources to Learn More
*Picture of Yellow Lab on Couch from http://www.blueskiesandyellowdogs.com
Written by Jared Knapp, EMC Atmos Principal Corporate Systems Engineer. Follow Jared at @EMCJared.
Some things are painfully obvious only after you see them for the first time.
As a 'wanna-be' inventor, when I see a new simple but clever invention, I chide myself for not having thought of it first. Like that sandal with the bottle opener embedded in the sole - I think to myself, "I could have thought of that!"
With age comes wisdom (sometimes…), and the truth is that it is easy to understand something once you've actually seen it, but it's not so easy to conceive of something you've never seen before.
This is what technology innovators are good at as well. They exploit a small change in the way we perceive a problem to create a whole new paradigm that substantially change the landscape. Sometimes the change can be so obviously necessary that we don't see how we ever got by without it before.
The Market is Finally Starting to "Get" the Power of Object Storage
In the storage world this phenomenon is being exemplified by the growing understanding of Object Storage's potential. In the beginning, a surprisingly long time ago, only the really smart people got it - probably because on its own, object storage doesn't do much. It's what it provides that gives it its value. And now that the platform itself is maturing and enterprise grade solutions have begun to be built around it, its benefits are becoming much more apparent.
Today, an increasingly distributed workforce is generating massive amounts of data that has begun to confound many traditional technologies such as tape, SAN, and NAS. IT Administrators are constantly struggling as they try to manage storage growth, balance storage capacity, and provide secure data mobility. The time has come for a new generation of file services that are properly able to handle these developing requirements.
EMC Atmos Object-based Cloud Storage Platform
EMC's Atmos is a multi-petabyte object storage platform that enable its partner solutions to address these issues and broaden our horizons. It does this by providing such things as innovative storage efficiency logic, geographic distribution of object content, and a unified global namespace. Its scriptable policy engine allows administrators to easily define where content should be placed, and what type of data protection should be applied. This means that DR is integrated into the solution, freeing administrators from the need for separate backup operations. The global availability of objects, and strong data protection provide a solid platform on which solution providers can build their products to deal with these issues of massive sprawl of unstructured data, and the requirements of a geographically distributed workforce to access it.
But an object platform is only a foundation to build up from. The true value of object is fully realized when you apply value-added, next gen services that harness the simplicity, scale, and efficiency of object. Here's how several of EMC Partners have easily onboarded their solution with Atmos to provide customers with simple solutions for file sharing:
Syncplicity's Sync and Share solution allows individuals to sync, access and share their files with anyone on any device. Their new Enterprise Edition has been recently released which features on-premise local storage to provide secure enterprise grade file sharing. By combining Syncplicity with EMC Atmos, Syncplicity has additionally provided multi-site active/active access to that data. This means that users in any geographic location can have their data transparently replicated with other geographic locations, and Syncplicity clients at those remote locations can access the data as if they were local to the source.
Panzura's global cloud storage system provides an enterprise-scale, globally distributed file system with military-grade encryption, unified namespace, global deduplication and global file locking. Using Atmos as their object storage platform provides the necessary global namespace, infinite scalability, and policy based replication and ILM to help create an extremely powerful, globally available, file system.
Maginatics MagFS is one of the newer solutions to partner with Atmos. Their hyper-efficient approach to cloud based file systems takes advantage of the power of modern devices to move their CIFS/NFS gateway functionality onto the client device. The MagFS client software can run on any device, including mobile clients, and provides native end-to-end encryption. Combined with Atmos storage MagFS provides cost saving massive scale, superior data protection services, automated data placement for efficient geographic distribution, and near instant access to datasets of almost unlimited size.
CTERA is a hardware and software based cloud storage enablement solution targeted at both enterprises and service providers. It is not simply a file sync & share and mobile collaboration solution, but CTERA also provides integrated cloud backup and cloud storage “on-ramping”, all managed through a single administrative interface. The solution takes advantage of Atmos’s uniquely architected services to scale to huge numbers of objects, manage efficient load balancing, provide integrated data redundancy, and support a secure multi-tenant environment. This enables the CTERA Portal service delivery platform to managing tens of thousands of end-points and sites within a cost effective, and easy to manage application.
These products all provide a different look at supporting exceptional next generation file services. By taking advantage of the opportunities of Atmos and Object Storage they are forever altering the way we do things, and what we can expect from file services in the future. As they change the landscape, we will look back and wonder how we ever managed without them.
Written by Mark O'Connell, EMC Atmos CTO,
A great partnership can make all the difference
I think we've all heard that cliche of searching for those possibilities where "1+1=3". While mathematically unsound, this does capture the true fact that when the right things are paired together, the resulting value can be greater than pure logic might suggest.
There are plenty of examples of this across all domains. What would Abbot be without Costello? When thinking of salt, pepper is always right behind. Or chocolate and peanut butter, which, as the commercials used to say, "two great tastes that taste great together."
In this blog, we'll take a look at two great technologies, cloud storage and object storage, and examine why these two technologies are coming together as the "marriage made in heaven", or, more colloquially, "two great technologies that go even better together."
Cloud storage is a new transformative technology, promising to fundamentally change the economics of the storage industry. When looking at cloud providers, either public or private, a few names stand out - Amazon, Atmos, Azure, AT&T Synaptic Storage, Cleversafe, Open Stack, Savvis, etc.
What's also interesting is what is paired with each of these - namely a full featured object storage platform which forms the core storage underlying the cloud. How this object storage was developed varies from company to company, with companies like AT&T or Savvis using Atmos to power their cloud, while companies like Amazon or Azure have home grown their implementations, while the OpenStack open source effort relies on the open source Swift object storage platform.
What is it that has driven each of these major cloud providers to base their cloud on an object storage platform? What is it about object storage that makes it uniquely suited for these environments?
According to the NIST, cloud has five essential characteristics: On-demand self-service, Broad network access, Resource pooling, Rapid elasticity, and Measured service. When looking across the different object storage systems in the market, a few key common characteristics emerge which are fundamental building blocks underlying the public characteristics of cloud.
Ease of growth: Because object storage systems abstract the underlying HW and SW, object storage systems identify objects in a way that is completely independent of the actual location of the object. This means that object storage systems have great elasticity to grow the storage and to allow applications to access the new storage without any reconfiguration or notification necessary. This also means that applications can organize the data in the most natural and convenient way, without concern for directory or path layout restrictions inherent in filesystems or layout restrictions on block LUNs. This allows for great elasticity to meet application demands for storage, and allows pooling of storage resources amongst many applications, where each application can utilize the storage in an on-demand fashion.
Scale-out storage: As object systems grow, both access and storage resources may be added. There is no need to have a separate access infrastructure which must be scaled separately to provide access to the storage. As object storage systems are composed of smaller, individual nodes which may be freely added to the system, and each invidual node provides both additional storage and additional networking capacity, growing the storage implicity grows the breadth of network access and further allows elasticity and pooling in the storage and networking resources.
Storage policies at a user defined level: As object storage systems abstract the underlying HW and SW, object storage systems can implement storage and replication policies at a level which is meaningful to a user, and users can determine the desired policies when provisioning the object storage and have the underlying object storage system implement it. In block or file based systems, the storage or replication policies are typically defined at the LUN or filesystem level by the system administrator, and these are shared by multiple users who then inherit the storage policies instead of being able to define them. This is a key element of an on-demand, self-service model.
REST based access: Object storage systems use the internet and mobile friendly HTTP based REST access patterns, which provide network access both across a wide variety of devices and across firewalls, in a safe and secure manner. This allows a level of network access far beyond that of a traditional storage system, allows for great elasticity in dealing with spikes in application network traffic, and allows pooling of networking and storage resources across multiple applications and use cases.
Multi-tenancy, self service, metering: Object storage systems typically allow users to have full administrative control over a slice of the system, with full rights to create new storage, and full rights to see their usage of the system (and also to be charged for their usage.) One example of this is the definition of storage policies by a specific user, while other examples would include adding new users to the system, setting quotas, or examining storage usage. This is also a key element of an on-demand, self service storage, as well as providing the core measured service which allows users to view, understand, and control their use of the service.
Metadata: Object storage systems typically allow tagging data with arbitrary metadata items, allowing the application to classify the data in a standard way, or allowing different applications to process the same data set in multiple ways. While some filesystems support user defined metadata on a per-file basis, this is not universal, and most filesystem based applications choose to embed such metadata in the file in an application-defined format, thereby limiting its cross-application usefulness.
The ideas of sharing a common storage infrastructure across users in a cloud fashion is not new - arguably the advent of multi-user systems, timesharing in mainframes or Unix systems, etc were all early attempts to achieve this goal in some fashion. However, it was not until the advent of object storage systems, with their unique characteristics, that the ideas of cloud could reach their full potential.
Just like chocolate and peanut butter came together to bring out the best in each other, cloud and object storage are truly a match made in heaven, with each bringing the other to its full potential, and the combination revolutionizing the storage industry.
Learn more about Atmos and the relationship between Object Cloud inthe Tenaja Technology in Brief: The Object Revolution.
*Abbot and Costello Image from crazy about TV http://www.crazyabouttv.com/abbottandcostello.html
Written by EMC resident #cloudnerd, Jason Cwik, Atmos Software Engineer, @jasoncwik
Along with the release of Atmos 2.1, the SDKs have been updated to give new features for our developers! Some highlights include:
Completely revamped SDKs in Java and C
Anonymous access tokens
Content-Disposition support in Shareable URLs
New checksumming algorithms
If you are upgrading to Atmos 2.1, be sure to download the new Programmer’s Guide
on the Atmos Developers’ Network
The Java SDK has been completely overhauled and now uses the Jersey framework to simplify the SDK and allow for richer interactions. For example, the readObject response used to simply return you a byte array or InputStream with the object content. While this worked fine, it discarded some extra information available in the responses like metadata, content-type, and ACLs. The new interface wraps the response in a response object that gives you access to all this information. If you’re currently making two calls to Atmos to get an object’s metadata and its content, you can now consolidate these calls into one. See the new interface in com.emc.atmos.api.AtmosApi for details. There are many other changes to the Java SDK. See the new quickstart on the Atmos developer community, or download the SDK and get started!
The C SDK has also been completely overhauled. It now uses an object-oriented (non-C++) framework that provides much more functionality and better memory management options than the old SDK. The new SDK also allows for content streaming, reusable content buffers, and has advanced hooks to fine-tune the underlying request structure and/or CURL handles. The code has also been updated to use GNU automake for better cross-platform support. There’s a new quickstart guide to help you get started and the SDK can be downloaded from Google Code.
The biggest new feature for developers in Atmos 2.1 is the Access Token. Access Tokens give you many more options for sharing content outside of Atmos. They also allow non-Atmos users to upload content. When you create a token, Atmos generates an opaque identifier and associates a policy with the token constraining its use. Once created, a token can then be used for downloading or uploading to Atmos.
You can create a download token for either an existing ObjectID or ObjectPath. The following policy settings are available for download tokens:
Defines the point in time that the token will expire. After this time passes, the token becomes invalid.
24 hours from create time.
Limits the number of times the token can be used for downloads.
Defines a set of IP address ranges that can be used to access the content. Both inclusion and exclusion are supported.
None (any IP can download)
You may also create an access token for uploading content to Atmos. This functionality provides a convenient feature to allow external applications or HTML pages to upload content directly to Atmos without exposing your credentials.
When you create a token for upload, you can specify a filename or directory name in the Atmos namespace for upload. If you don’t a simple ObjectID will be created. Like download tokens, a policy is associated with the token and has the following options:
Defines the point in time that the token will expire. After this time passes, the token becomes invalid.
24 hours from create time.
Limits the number of times the token can be used for uploads. The only valid value here is 1.
Defines a set of IP address ranges that can be used to access the content. Both inclusion and exclusion are supported.
None (any IP can upload)
Defines the minimum and maximum sizes for the uploaded content.
None (any size is acceptable)
Places constraints on the fields that must be present in the HTML form used to upload the content.
None (any form fields may be set)
When using the upload token in object mode, the created ObjectID will be returned in the “Location” header of the upload response. Optionally, you can set the x-emc-redirect-url in the HTML upload form to have Atmos redirect the browser to another page upon completion of the upload. This page can capture and process the upload response. You can also set other headers in the HTML form like x-emc-meta to set metadata on the new object. There’s a lot of functionality available here; too much to cover in this blog post! For more information please see the Atmos 2.1 Programmer’s Guide.
Content-Dispositions in Shareable URLs
The shareable URL functionality in Atmos now also allows you to specify the value of the HTTP Content-Disposition header when downloading content from Atmos. This allows you to fine-tune how web browsers will handle content served by shareable URLs. For example, if an object’s Content-Type is image/jpeg, a web browser will display that content in the browser window (aka, “inline”). If you really want the browser to download the file, you can set the Content-Disposition header to “attachment” and the browser should download the file instead of displaying it. Conversely, if you have a format like PDF that browsers might want to download and you want to embed it in the page, you can use the Content-Disposition header to “inline” to suggest the browser display it instead of downloading. There is also functionality to suggest a filename for an object when downloading. This is especially helpful when serving a shareable URL from an ObjectID. Since the URL does not include a filename (only the ObjectID), you can now set the Content-Disposition header to include a filename for your ObjectID, e.g. “attachment; filename=myimage.jpg”. This functionality is governed by RFC 2183 and RFC 6266. Browser support varies (especially for UTF-8 name extensions), so cross browser testing is suggested.
New Checksumming Algorithms
By popular demand, we have added new checksumming algorithms to the SDKs. When uploading using the “x-emc-wschecksum” feature, you can now select SHA1 or MD5 in addition to the existing SHA0 algorithm. Using the new algorithms are simple. For example, in Java you simply request the new algorithm when constructing your Checksum object:
RunningChecksum wsChecksum = new RunningChecksum( ChecksumAlgorithm.SHA1 );
CreateObjectRequest request = new CreateObjectRequest().content( data ).contentType( "text/plain" );
request.wsChecksum( wsChecksum );
CreateObjectResponse response = this.api.createObject( request );
Get Started Today!
Atmos Online has already been updated with all of these new features. If you haven’t already, sign up for an account and visit the developer site to download the new SDKs and samples!
Last week's blog and voting contest focusing on a 2012 retrospective gave us an opportunity to step back even further and look around at some recent analyst views on the industry that closely match what we see in our install base.
The signs of storage transformation practices are everywhere. Within these, three areas emerge starting with unstructured data (specifically files) how files are served to users, and what users do when when IT can’t provide what they need.
Trend # 1. Where's unstructured data coming from?
According to a recent ESG study, the top source of storage challenges comes from the rapid growth and management of unstructured data.
Trend # 2. Where does the data need to go?
According to IDC, the death of the workstation continues as BYOD trend gains even more momentum. Here we see that for the third consecutive year, smart phones and tablets combine to claim more share of the 'smart connected device' market and now account for more than half of the market.
Trend # 3. What happens if/when users can't get to their data and files?
What happens when IT and the storage team doesn't give users what they want? What will they do to get access in their quest to stay connected to the network, to their customers and to their content?
If you guess that they go around IT firewalls to say in in sync – you are right! This pie chart below shows the continued trend in shadow IT usage. Why? Because people 'gotta have their stuff ', and in this case, more than half of them surveyed won’t take "No" for an answer.
Cloud Storage – The Signs of our Times
Content, Content, Content. You have to work to manage it, why not manage it differently to make it work for you?
Sign # 1: Use a Platform for Unstructured Data
As evidenced by the mobile device profilieration in today's workforce, besides access and sharing abilities, one aspect of mobility that file system protocols cannot overcome is that of latency and distance. Object storage systems solve these challenges in two ways: Native REST and Eventual Consistency both of which are built into the Atmos architecture, not bolted on via some other app.
REST is WAN and firewall friendly and inherently more highly available and resilient to network interrruption than legacy protocols. Before you had a mobile device, when you were on the road or at home, and needed something in your home directory or a network share...what'd you do? Chances are you VPN'd in, mounted a drive, and (slowly) read a file. That trip over the internet back to you is VERY expensive and chatty. REST solves this in the datapath by being very smart on the WAN and able to handle the communication between user and system "statelessly" with a single Create or Read, command. Don't believe me, watch the slideware examples HERE
The final benefit is the nature of distributed storage versus clustered file systems. Storage clouds are fundamentally different from filesystems in their architecture.
For instance, ten years ago (in what has become known has the CAP theorem) it was rigorously proven that it is impossible for a distributed storage system to simultaneously guarantee data consistency, data availability, and partition tolerance.
With these theoretical considerations firmly in mind, modern storage clouds have been architected to support the notion of "eventual consistency" and thus cloud storage platforms can make their stored content always available, anytime, anywhere.
This sort of massively-scalable information availability is something that traditional filesystems simply cannot provide. Read more about it in the architecture white paper.
Sign #2 : Provide native support for mobile applications and access
Considering building on your own? use the SDK
More and more enterprises have the ability to build their own mobile apps. For them, Atmos has an active developer community, a wide-range of and examples of custom mobile apps in both IOS and Android. These comprise a Software Development Kit (or SDK)
Here are 3 examples of Atmos-Powered apps from the SDK Lane:
EMC IT - Taking a Mobile First approach
The File Store Custom Mobile App. This app uses IOS on the client side in a secure container, and uses Atmos in the back end.
The app is a mobile front-end for home directories and personal files, available to EMC users worldwide, and leverages a multi-site active/active instance of Atmos in 2 data centers in the Eastern US. It also leverages the Atmos management API that automatically creates a subtenant/UID pairing, matches it to a user on the fly, enforces a quota, without ANY involvement from IT.
It's storage as a service in seconds.
A more detailed inverview with KK and Chuck HERE
EMC Mobile Marketing:
EMC Folio and EMC Mobile
20,000 + users and going strong. Two native IOS-based mobile apps, with rock-solid performance and worry-free scale, both back-ended by AT&T Synaptic (powered by Atmos and ranked as a market leader by Gartner). You can learn more about the business case and architecure of mobile here
What about Android?
Looking for something off the shelf ? That's the plug & play lane for mobility. Here, someone else has already written the code and built the integration.
With these apps, it's important to consider how they connect, and how they stay in sync, and how file locking is managed. First, let's look at sync versus shove.
Sign # 3 - Consider the Sync and Share Behavior and Frequency.
In the Sync & Share path, Syncplicity is the most likely choice. Since it let's you sync data in place from its current location (rather than picking it up and dropping it somewhere before it can be shared) this will push actual sync across all your devices.
Along with robust IT controls for administration and management it's the "killer app" to control that shadow IT behavior ESG captured in their survey.
For more about Syncplicity and Atmos, watch our recent webinar
It's important to distinguish between full sync versus offline access or more broadly, the category of "access solutions". These are apps that act as gateways or web-enabled front-ends for back-end file systems. Need to get data from your home directory, log in to query it. Want to share with a colleague? You may have trouble syncing that content across all your devices. These access solutions often don't have REST (which supports async reads & writes) as a result, when you need to "share" with your team-mates, its a shove and share, versus a sync and share.
Access soutions are also considered "gateways" since (under the bonnet) they're translating protocols from REST to CIFS/NFS or vice versa.
Gateways come in different flavors too.
Here are 3 apps with a wide range of features, admin controls and pricing options.
For those looking to offer multi-user access between the desktop and single server (via CIFS/NFS), where that server then communicates via HTTPs/REST to the cloud, Atmos GeoDrive satisfies this use case.
Plus, its free..and free is good.
Other more advanced gateways offering on-ramps to the cloud are Panzura with their unique distributed file-locking capability to deliver a global, multi-site NAS experience for local users, all back-ended by a single cloud repository.
Seven10 presents an enterprise-wide archive interface for legacy apps and users needing basic file sharing capability, as well as a seamless approach for data migration.
A closing thought:
Free Your Mind about Mount Points and File Services
If you're faced with a BYOD or other mobile initiative, now is a good time to 'free your mind' and think more strategically about file services in general rather than just replacing home directories. Just because a mount point looks like a mount point, to break free from traditional storage constraints of physics and latency, it needs to be connected to cloud storage.
Cloud storage (objects accessed via HTTP/REST) was once releguated to archive-only use cases but has steadily been mainstreamed to satisfy primary storage use cases in ways that traditional storage cannot.
Freeing your mind to cloud storage delivers the choice, efficiencies and economic models that traditional storage and LAN-based access struggles to deliver.
IDC Study "Mobility Reigns..."
It was a very good year.
2012 was a banner year for Atmos. We announced Atmos 2.1 with support for up to a 100PB cloud, new packaging and pricing options for greater flexibility, and S3 API support to migrate application data in and out of Amazon.
Plus the 480 Dense Hardware platform hit the streets, offering customers more capacity with less infrastructure and power. And the increasing demand for web and mobile applications finally hit file system’s tipping point and drove more enterprises and service providers to object-based cloud storage for its limitless scale, efficiency and access.
Our cloud experts learned a lot in 2012, and shared their knowledge, know-how and lessons learned with you through the Atmosonline blog. Now, a year wiser, 20+ blogs and 30,000 visitors strong, we wanted to revive and share the top 3 blogs of the year. These articles not only got the most views, but sparked the most dialog, questions, and ideas.
Share your feedback for a chance to win an iTunes giftcard or a #cloudnerd tshirt
Take a walk down memory lane, read the oldies but goodies, then share what you think on Twitter:
- Vote for your favorite
- Recommend the blog series
- Share your opinion or experience with Atmos
Be sure to tweet @EMCAtmos and include #AtmosCloud. Winners will be chosen at random and notified through direct message.
- #1 Blog of 2012 Panzura and Atmos: Rethinking NAS with Cloud Storage
Panzura is an integrated solution for global file share across distributed NAS that can store data in Atmos cloud storage. This article discusses how Panzura takes advantage of the REST API - in the data path - to store and archive data in Atmos Cloud storage.
- #2 Cloud Storage: What Indy Doesn't Understand About Modern Archives
This great play on the Indiana Jones theme, shows the Arc stored among thousands of boxes that look exactly alike. Sound familiar? Without metadata – your data looks exactly the same. This article explores the power of metadata, active/active archives, and instant access to data.
- #3 Cloud Storage: Why It's All About The App
Applications drive storage. And modern web and mobile applications demand Web services and RESTful APIs to abstract applications from underlying storage, provide instant access from any device, to break free from file system limitations. This blog is one of many that compares file vs. object storage for unstructured date – and comes complete with code examples.
A look ahead to 2013 and beyond
EMC has a long standing commitment to Object Storage Technologies. 11 years ago, EMC Centera pioneered the way for object and today continues to be the #1 object platform – delivering 99.9999 availability, content immutability and compliance. 5 years ago, EMC Atmos came on the scene to expand object storage with cloud attributes such as: active/active, global namespace, multi-tenancy, policies, instant access, and storage-as-a-service. In fact, more than 50 service providers power their storage services on Atmos, and hundreds of enterprises use Atmos around the globe to efficiently manage, store and archive data created and consumed by today’s content-rich web and mobile applications.
We expect the demand for object to explode in 2013, and our product solutions today as well as a feature-rich roadmap across EMC platforms will ensure your investment in EMC technologies will continue to add value for years to come.
Image credits from http://www.youtube.com/user/mukke67?feature=watch
It wasn’t that long ago when most of EMC product information came to you in a document, or more appropriately LOTS of documents. REALLY BIG DOCUMENTS.
- Want to learn about new products? Go find and read the specs.
- Want to read up on a new feature? Go find and read the manual.
- Want to upgrade or install a new service pack? Get the manual supplement.
This was a daunting task just to keep up.
Mobile and Cloud changes everything:
A few years ago all that changed when mobile devices eclipsed laptops as the preferred way to access information. And, like many organizations, EMC had to shift their mindset, applications, and storage to interact with customers, instantly on their device of choice. To provide real-time information about their products they used today, and ways to get more value of out of their investments in the future.
Enter the cloud.
EMC formed a Mobile Marketing team. Headed by Randy Ziegler, their mission was to create a purely-mobile experience for customers and EMC Ecosystem partners. They took a phased approach for success. Their initial requirements were to:
- create a native mobile app (based initially in IOS on the iPad)
- get it to market quickly
- scale up to support thousands of users worldwide – available to anyone, anywhere who uses the Apple App store
- keep the content up to date, engaging and visually appealing
- build it with minimal IT infrastructure investment
- and finally, not worry about the storage platform or mundane tasks of provisioning, protecting, scaling, availability, etc.
However, as they began building a completely different type of application and user experience, they soon ran into two unforseen issues related to their web portal-based and small-screen smartphone apps:
- The search/filter/download interaction of a web page squeezed into a tiny mobile device screen is a poor user experience compared to a client-based app.
- For optimal performance and scale, using an HTTP/REST method to distribute content from object store was the right architecture.
It was time to take a new approach, one that centered on RESTful APIs and Cloud. Why? Because the application architect Aashish, was already familiar with the REST API for Atmos after building his own sample apps a few years prior as one of the first participants in EMC's developer community. He knew the tool set, but now needed a development environment and platform. For this, they chose AT&T Synaptic, (see Tech Specs Tab) powered by Atmos cloud storage technology.
Time to market:
From these lesson's learned, EMC Mobile introduced their EMC Folio, their first moblie app, at EMC World in 2012. Customers were so blown away by the user experience, ability to collaborate with EMC, and wealth of information at their fingertips, that it rapidly grew to over 15,000 users in just a few months.
And, through Folio, EMC could understand user behavior and start to tailor information to customers based upon their role in the organization. So, a year after Folio as they planned their next release, called EMC Mobile, the development team, had a head start. Since the API commands that both Folio and Mobile use are similar (Create, Read, Update, Delete) the effort needed to build, test, deliver, and scale a new app is very small. Such is the case with apps built on an API.
What was the result?
- A small team, with bright ideas yet limited resources, facing pressures of time to market. Sound familiar?
- Two very interactive iPad apps, using a public cloud infrastructure, and a developer-friendly API, that delivers a consistently high-performing user experience for any user, anywhere in the world. Sound impossible?
It's not. Listen to Randy’s full interview here.
Behind the Scenes Architecture: Benefits of HTTP/REST and Cloud Storage:
Aashish Patil, architect and product manager was a key player in the technical success of these applications and was one of the early founders of the EMC Developer Community. Aashish was very familiar with building applications on the Atmos API and its Objective-C wrapper for the IOS operating system on the iPad.
The Atmos API made the application easier to code and operate from a mobile device. Here's way. Rather than processor-intensive searches of file systems to locate content, objects are stored in the cloud to avoid having to push all content to everyone’s mobile device. Once objects are stored in the cloud, they’re then referenced by a unique Object ID which lives as a metadata tag attached to an image of that content, much like the cover flow are you might see in iTunes. When the user sees a piece of content they find interesting, like a data sheet, or customer interview video, they simply tap on the picture and the app calls the object store to retrieve the actual object, which is then sent via HTTP/REST back to the device.
I had the opportunity to spend a few minutes with Aashish recently where he whiteboarded the architecture, the data flows and the positive experience building a mobile app on an Atmos-powered cloud.
Here's a video fo Aashish whiteboarding the architecture:
EMC attributes it's 3-year success, with 2 custom applications that serve 20,000+ users to:
- Eliminating the need to ‘think about the back-end storage’ by using AT&T Synaptic, a public cloud service recently ranked by Gartner as one of the top 5 globally
- Worry-free performance of a multi-site active/active cloud storage service from AT&T Synaptic that delivers a consistent user experience world-wide.
- Service Provider Ecosystem. As Randy says: “It just couldn’t be easier to set up an account and create a development environment on Atmos. Does it make it easier? Absolutely.”
Check EMC Folio EMC Mobile at the Appstore Today!
Want to try the API yourself? Try the Atmos API simulator and sample 5 common commands. You'll automatically be entered into the drawing for a #Cloudnerd shirt like Aashish's and many others who have also built web or mobile apps on Atmos worldwide.
Chances are, now that January has come and gone, you've probably conducted that annual ritual of pulling out, using, and putting away your holiday stuff.
And although, as some holiday songs claim, "It's the Most Wonderful Time of the Year" if you listen closely to the lyrics, you'll notice they NEVER talk about the hassle and effort to move decorations to and from the attic, or the or closet, garage, basement, storage locker or wherever else you've hastily shoved that stuff since last winter…
Unused data is just like holiday decorations
You use it for a short period of time, stow it away, and retrieve it when you need it. You wouldn't:
- Throw away all your "stuff" and buying all new "stuff" every year? Not very viable.
- Leave everything up in a prime location like you living room? - all year long? No way.
- Pay someone to move everything around? Would be nice to sit back and watch, but also not very viable economically.
So why would you do the same with your unused data? While we can’t automate the holiday process, we can automatically tier data to low cost cloud storage with instant access whenever you need it to retrieve it.
EMC Cloud Tiering Appliance Automatically Tiers Unused Data to efficient Cloud Storage
So while you grudgingly move your "stuff" around to optimize prime space (think: the living room) every January, you’ll be happy to get back to the office, where automation and point-and-click simplicity make your job a whole lot easier. EMC Cloud Tiering Appliance and EMC Atmos gives a unique one-two punch that delivers measurable business value to:
- Reclaim capacity on Tier 1 NAS (EMC Celerra, VNX or NTAP Ontap Filers) often the source of the unstructured content sprawl.
- Overcome networking and domain restrictions
Most enterprises have limits on the ability for CIFS/NFS protocols to communicate freely on the same LAN, or even over great distances across the WAN. To solve this problem with ease, CTA communicates via a firewall-friendly HTTP protocol rather than the chattier, latency-sensitive LAN protocols.
- Transform your business model
Since CTA is integrated with the Atmos REST API, it can communicate (or tier) content to a local instance of Atmos in a Private Cloud, and/or tier content to an off-site 3rd party Public Cloud such as any of the 50 service providers world wide running Atmos. Another parallel exists between how you store decorations at home and content at work that follows Private Cloud (CapEx) and Public Cloud(OpEx) economic models: Depending on my economics, I have the choice to store my holiday stuff in my home attic (which I bought + maintain) vs putting my stuff in some self-storage locker across town (which I pay a monthly utility fee for). Leveraging the encryption in CTA, the WAN and firewall-friendly HTTP access to the cloud, you have economic possibilities that don't exist when tiering from one NAS tier to another via CIFS/NFS.
- Deliver a transparent user experience at the desktop
CTA is fully integrated with NAS, so users still see their same file shares, mapped drives and folders they're used to seeing. When (or if) they actually want to go back to open an old file, a simple double-click calls a flat file (or stub) that points to and pulls their file from the cloud to the desktop.
Storing unused data in another tier (like the attic or basement) is a strategy that works at home…and in the office.
Keep it or Kibosh It?
In my house, there can be times of conflict fueled by a sense of separation anxiety when deciding what should be kept and what should be actually, finally, thrown away or donated.
Letting go of content is tough. No matter how old it is. You just might need it some day.
Truth is, I usually lose this battle at home since a) the perceived "value" of an item is pretty subjective and b) there's always, always more content sprawling its way into the house on the next birthday, Christmas, school year or whatever.
To some extent, the same dynamic exists in our digital world and work environment, but the rate of newly created or acquired content is greater, and in some cases, much, much greater. And although at home I may reluctantly agree to throw content away (think: Kibosh) at work this is hardly, if ever possible in our compliance-driven, data-driven, Big Data world.
Here are a few customer examples that discuss how EMC CTA and Atmos deliver complete automation, data flexibility and access, scale and economic benefits:
Brian Dearman, Systems Analyst, University of Illinois Hospital:
Retrieval times from Atmos are so fast, there's barely a difference compared to VNX. Before, retrieving archived medical records from tape took a week. Now, users don't have to actually retrieve data before burning a CD, they're retrieving data at the same time.
Chris Wigglesworth, Head of Host Systems, Information Technology, The Open University (UK)
Using EMC Atmos storage in conjunction with EMC VNX and CTA has allowed us to reduce our storage costs to a third of what they would have been if using VNX alone. The EMC Atoms storage was quick to provision and is simple to administer. The REST interface is easy to understand and use. The policy engine offered by CTA is flexible and feature rich allowing it to solve a number of different storage scenarios in our organisation.
Edward Poll, Systems Manager, IT Department, Cranfield University (UK)
Cranfield University IT had a requirement to extend its usable NAS based storage, whilst alleviating the pressures around backup. By implementing EMC Atmos, alongside the existing EMC Celerra and EMC CTA infrastructure, we were able to increase our available storage 10 fold and at the same time reduce our backup footprint. With EMC Atmos’ flexible object based replication policies and it’s highly available hardware architecture, we were able to implement a cloud level storage appliance to meet all our needs, now and longer term.”
So now that you've put away all those decorations, "keep momma happy", and reclaimed the prime resource of space in your living room, now 'tis the season to do the same reclaimation process for your NAS resources.
Learn more about EMC Cloud Tiering Appliance and EMC Atmos
Guest blog post by Rich Radin, Sr. Business Development Manager for Service Providers, EMC.
Every day we are reminded that EMC Atmos is not the only cloud storage game in town. Enterprises have many choices in the market – and a healthy dose of competition is what provides the Atmos team with perspective to drive innovation, creativity and focus on success – no matter where we sit on the Atmos project team.
So where do we sit? How do we stack up? And what can you expect from us?
Let’s start with the new players and how they’ve changed the game. The names certainly have changed, with newer entrants such as Amazon, Microsoft, Google, Nirvanix among others – that aren’t your typical service providers. They capitalized on the prevalence of web and mobile platforms that demand Web services and instant access to storage. And they have responded with a whole new delivery method that offers swipe and go, commoditized storage services over the Internet. It’s a brave new world.
So, how does Atmos fit into this model?
We contemplated this game, and realized that our value lies in the ability to empower and help those who want to get into the game, play it well, differentiate, and win. So instead of offering storage services ourselves, we’ve poured all our energy and power into making our partners and their developers (who represent the biggest names in telecom, hosting, and Systems Integration, and are better suited to deliver bits over wires) with a scalable cloud platform, rich SDK, and Web services so they can rapidly build and deploy many differentiated services to market faster.
How do we stack up?
Recently Gartner accessed the critical capabilities of current cloud storage providers across a number of vectors. These included accessibility, manageability, cost, resiliency, security, compliance, as well as value added services such as ISVs, industry specific solutions, and SLAs. One of EMC’s top partners, AT&T, powers its Synapatic Storage-as-a-Service offerings with EMC Atmos, and was among the top providers. AT&T not only bested well known brands, such as IBM, HP among others – it was on par with Amazon and others.
Link to the full Gartner report
Though the Gartner report was limited to major, mostly U.S. based providers in the industry, it’s worth mentioning that EMC Atmos and Atmos Cloud Delivery Platform power over 50 public storage clouds globally, including the likes of China Telecom, MTI, Ninefold, Savvis, Swisscom, Telecom Italia among many others.
Why is this so important? Global reach is not local reach. Atmos enables in-country delivery and regulatory management of storage on every continent we serve, which includes them all. These xSPs have physical, personal, and locally aware relations with their customers. The same Atmos underpinnings that earned AT&T recognition in Gartner’s public cloud evaluation is the same core kit used by all our service provider customers.
It’s important to note that these solutions are backed by the number one name in Enterprise Storage, EMC. And, many of these service providers are part of the EMC Velocity Service Provider program, which drives partnership and cooperation from EMC customers to Service Provider’s completeness.
Are you ready to differentiate your business with cloud services? Do you want to partner with a trusted storage market? Look no further – like AT&T and 50 other public services powered by Atmos – we can help you deliver new differentiated services to compete, protect, and take more share in the market.
Here's where you can find answers to help.
AT&T Synaptic Storage, Powered by Atmos, Praised by Gartner
Many Faces of Atmos - Solution Profile Video Montage
Security Risk of Consumer Tools in the Enterprise
Consumer file share services like Dropbox or GoogleDrive are so easy and pervasive that employees are going around IT and using them to save, access and share files that are just too big to pass through Email or iPhones.
In a recent survey, Symantec Survey Reveals Online File Sharing Poses Great Security Risks, employees were asked how they would go about sharing a big file, they responded:
“Ask IT for help (51 percent), use a solution suggested by a customer, contractor or partner (42 percent), utilize the IT system in place (33 percent), or search online and download a free solution (27 percent)".
To address these trends, IT had few choices to meet user requirements while retaining control and data protection. They could build it, block it, or buy into it. Now, with the recent announcement, EMC Syncplicity for Online File Sharing with On-Premise and EMC Atmos Object-based Cloud storage, enterprises can finally provide it.
IT Rejoice: Provide Users with Online File Sharing, Keep Data on Premise
With the new on premise Atmos storage option for Syncplicity, EMC can offer enterprises the best of all worlds - a cloud-based online file sharing offering that simplifies end user deployment and administration with on-premise storage that gives IT control of the storage layer, in addition to continuing to provide Syncplicity storage in the cloud.
The benefits of combining EMC Syncplicity online file sharing with EMC Atmos include:
- Increased Productivity: Users can easily sync, access and share files with anyone, anytime, anywhere and on any device for new levels of productivity and agility while giving IT the security, visibility and manageability they need.
- Flexibility and Ease of Management: Enterprises will experience the agility and continuous innovation from deploying a cloud-based online file sharing solution for users while retaining complete control over data and storage resources. It combines the rich set of Syncplicity’s administrative tools and security, compliance and policy controls to manage sync-and-share functionality along with the unmatched manageability, scalability and resiliency of EMC storage.
- Reduced Compliance Risk and Increased Control: Enterprise information will reside on EMC storage, situated on-premise, subject to IT security governance and protection policies. Files are not duplicated both on-premise and in the cloud. With the EMC solutions, data objects stored on premise remain on premise and within IT control.
Why Atmos Object-based Cloud Storage Behind Syncplicity?
EMC Atmos provides industry leading object storage technology, designed to support large multi-site, multi-tenant, active-active environments. Atmos allows the application and storage to run anywhere and provides metering and chargeback capabilities based on bandwidth and consumption to ensure IT retains the necessary controls.
Watch the Interview with Jeetu Patel GM, EMC Syncplicity Business Unit and Ondrej Hrebicek, CTO, EMC Syncplity Business Unit as they discuss how the Syncplicity and Atmos solution:
- Users can easily sync, access and share files with anyone, anytime, anywhere and on any device.
- Combines the rich set of Syncplicity’s administrative tools and security, compliance and policy controls to manage sync-and-share functionality along with the unmatched scale, self-service and integrated application.
- Geographically place content on an active/active, scale-out object store.
- Leverage secure multitenancy, allowing a single cloud infrastructure to support additional business initiatives.
Learn more at the scheduled Webcast, on Thursday, February 7th, 2:00 PM ET.
or read the solution overview