使用Peerislands Mode亚博贵宾会贴吧rnation Tool Set迁移到AWS上的MongoDB Atlas
随着云计算在整个行业中变得司空见惯，组织正在迅速采用MongoDB地图集，因为他们知道真正的现代化不仅仅是将数据按原样移至云，即亚博贵宾会贴吧采用“举升和转移”方法。这也是要重塑相同的数据，以更快，更迭代的开发。借助Mon亚博贵宾会贴吧goDB的基于文档的数据库，开发人员有权通过灵活的架构设计重新构想他们如何构建他们的构建，从而使他们可以轻松地为广泛的用例建模和重塑数据，同时仍然在需要时应用治理。亚博贵宾会贴吧MongoDB地图集自然地映射到现代面向对象的编程语言，使开发人员的生活变得更加轻松。与SQL数据库的刚度相反，MongoDB的灵活数据模型意味着您的数据库模式可以随着业务需求而亚博贵宾会贴吧发展。这可以帮助用户更快地构建应用程序，处理多种数据类型并更有效地管理应用程序。作为一项完全管理的服务，MongoDB地图集为您负责数据亚博贵宾会贴吧库维护，并且还可以在多个分布式数据中心内部和范围内缩放，从而提供了以前无法使用关系数据库可用性和可伸缩性的新级别。搬到MongoDB Atlas的优点很明显，但是一些公司亚博贵宾会贴吧可能仍然不愿留下他们熟悉的未知领域的遗产关系数据库。这是Peerislands进来的地方。有了Peerislands，您不必一个人去。 The following blog introduces PeerIslands’ modernization capabilities, and how you can leverage them to migrate seamlessly to MongoDB Atlas on AWS. Why PeerIslands? PeerIslands is an enterprise-class digital transformation company composed of a team of polyglots who are comfortable across multiple technologies and cloud platforms. As a services firm, PeerIslands is focused on helping customers with both cloud-native development, and applications transformation. With best-in-the-industry talent, the team has helped several Fortune 50 companies bring large-scale transformations to life, and has received recognition from several clients and partners, including MongoDB. With engineers trained and certified in MongoDB, PeerIslands has helped MongoDB’s ISV and retail customers modernize, moving software built for on-prem to SaaS environments more conducive to cloud environments, and was named MongoDB’s Boutique System Integrator Partner of the Year . PeerIslands can swiftly transform and migrate core, legacy, and on-premises applications to the cloud. They develop solutions based on cutting-edge microservices and serverless architecture across public cloud platforms and hybrid PaaS platforms to help users quickly get applications to customers and business users. How PeerIslands can help PeerIslands has been working with MongoDB and AWS to develop tools that address two key objectives for customers: Objective 1: Tools that address common customer questions when evaluating MongoDB MongoDB Test Data Generator: A fully UI-driven tool with an extensive data library for rapidly loading MongoDB with use-case specific, near real-world data at scale MongoDB Performance Testing tool: A performance analyzer where you can create multiple load profiles, run-use case specific MongoDB queries and understand the performance of the queries. With the test data generator and the performance testing tool, customers can get a clear view of the performance of MongoDB for their specific situation even before migrating to MongoDB MongoDB Schema Generator and Data Modeler: SchemaGen tool helps to rapidly generate draft JSON schema from your existing SQL schema. On top of this, you can then perform the data modeling exercise and generate schema to form your MongoDB schema. The schema generator also provides key information about the SQL DB like size, index, and more MongoDB Sizer: MongoDB sizing tool helps you understand the size implications of your schema and calculate Atlas sizing. With the MongoDB sizer, customers can upload their own schema and calculate the various factors that influence the Atlas sizing Codescanner: A tool for scanning your code repositories for deprecated MongoDB APIs. With the code scanner, customers can get a clear view of the application impact for upgrading MongoDB versions Objective 2: Tools that accelerate time to value by rapidly moving workloads to MongoDB COSMOS2Atlas migration: A point-and-click solution that helps COSMOS customers migrate data from COSMOS to MongoDB. This solution provides change capture capability to ease downtime requirements and makes data migration easy and seamless 1Data: A tool for addressing more complex requirements of migrating data from SQL to MongoDB Admin mobile app: A mobile app for admins to track key Atlas KPIs and approve common access requests on the go PeerIslands brings to the table an entire suite of tools for addressing all your MongoDB needs. PeerIslands use-case featuring 1Data tool One of the key requirements of modernization projects is to solve large-scale data migrations from SQL databases. There are a number of tools that are available which simply replicate data from SQL to MongoDB—but, we rarely use the same SQL schema in MongoDB. Schema transformation—however difficult to do at scale—is nonetheless required so that we can make the best use of MongoDB capabilities. Today, the typical approach is to run custom Spark jobs as they are scalable and flexible when it comes to processing schema transformations and loading the data into MongoDB. But when you go beyond migrating one or two tables in a Proof of concept (PoC) setting, the problem becomes much more complex. For instance, writing custom Spark programs for every schema transformation is cumbersome and error-prone. For even simple migrations we will have tens of Spark programs. Any defects that occur during transformation are going to cause significant issues. Also consider the following challenges: How do you extract data out of your SQL database without impacting database performance? How do you handle infrastructure provisioning and scaling? How do you orchestrate the migration? Few master tables can be migrated once but transaction tables may need both one-time migration and a daily incremental migration. How can you do this orchestration at scale? How do you know whether you have not lost data during migration? Last but not the least, once a data is migrated how do you keep it up to date? We will probably end up with a suite of tools to address these issues–SQOOP, Kafka, Spark, some kind of a job orchestration engine, an observability suite, notification workflow and so on. It will quickly become evident that migrating data from SQL to MongoDB without disrupting business could be the most daunting barrier to adopting MongoDB. Unfortunately, current tools invariably fail for complex heterogeneous migration scenarios and developers end up writing a lot of custom code. Realizing this issue, PeerIslands has been working with MongoDB and AWS to develop 1Data. 1Data is a platform that helps enterprises perform migration and real time synchronization of data between SQL databases and MongoDB. 1Data is designed to complement existing AWS services like DMS in migrating data out of SQL. Key features of 1Data: Data is fully GUI based — There is no coding required 1Data provides a single platform for both one-time migration and continuous updates 1Data is consistent across one-time migration and continuous updates. This provides a good anti-corruption layer for continuous updates The tech stack of 1Data is based on Spark, Kafka among others and is highly scalable 1Data is highly modular and has a well defined API layer. 1Data can be easily extended to your needs 1Data automatically handles all the infrastructure required for migration with AWS quick start templates High Level Solution Architecture 1Data capabilities are realized through a decoupled and highly scalable architecture. The data extract, transformation and load part are independent of each other and can easily be customized based on the specific requirements of the customer. The architecture can orchestrate between batch-based initial loads and streaming-based CDC loads. A Spark, Kafka, and Airflow-based tech stack provides excellent scalability for the 1Data platform to handle large data migrations. Figure 1: 1Data High Level solution architecture OneData Portal structures migrations using Endpoints, Tasks and DAGs (Directed Acyclic Graphs) Endpoints define source, intermediate and final data locations and can come in the form of files, databases or queues. Endpoints can also be database extracts in S3 from AWS DMS service. Task definition is the second step in the migration. Tasks act on source point and produce data in either staging or destination end point. There are a number of predefined tasks available:Extract, Transformation, Sink and Validation tasks. You can configure both streaming and batch tasks. Defining the DAGs is the final step before actual migration. DAGs are used to define the sequence in which a user wants to execute the defined tasks. The technology components used in 1Data allows for easily handling very large data migrations. Each of the components has been selected such that they can be deployed across multiple cloud platforms and can be scaled easily. Technology Stack details below: Web Portal: Angular WebAPI: Node Configuration Database: MongoDB Data Transformation & Validation: Spark Data Extraction: Sqoop, Spark, DMS Change Data Capture: Kafka, Debezium Data Sink: Spark Job/Task Orchestrator: Airflow PeerIslands has worked with AWS and MongoDB to create a Quick Start for 1Data. With Quick Start, customers can rapidly instantiate 1Data for their migration requirements. To recap, with 1Data Quick start on AWS, we can Perform heterogeneous schema transformation from SQL and load data into MongoDB Atlas on AWS Weave together continuous data updates, incremental data updates and one-time migration using a combination of batch and streaming jobs Orchestrate the migrations tasks Validate the migration ...And all without writing a single line of code! Demo Looking forward A modern, data architecture can help you unlock your business’ full potential, and gain real-time access to the insights you need, when you need them. MongoDB’s document-based database and flexible schema design help you make smarter decisions, cut costs, and take full advantage of AI/ML capabilities to empower your employees and raise customer satisfaction. The decision to migrate off your legacy systems and onto MongoDB is easy—and now the process is, too. Let PeerIslands help you get there. Our best-in-class teams leverage next-generation technologies, including Artificial Intelligence (AI), Augmented Reality (AR), Blockchain, Internet of Things (IoT), Machine Learning (ML), Mobile, and Virtual Reality (VR). Our expertise spans the modern programming stack, and we follow best practices in distributed, agile, and lean principles as well as test-driven development and DevOp. Additional Resources ISV WMP Program Contact aws-isv-workload-migration@amazon.com for details Atlas Quick Start MongoDB Atlas Starter Package Atlas Migration Guide Atlas Migration Pattern Contact us with any questions around modernization with MongoDB, AWS, and PeerIslands.
2021年10月28日