Setup python environment using anaconda and Jupyter

Anaconda is the leading open data science platform powered by Python. The open source version of Anaconda is a high performance distribution of Python and R and includes over 100 of the most popular Python, R and Scala packages for data science.
Additionally, you'll have access to over 720 packages that can easily be installed with conda, our renowned package, dependency and environment manager, that is included in Anaconda. See the packages included with Anaconda and the Anaconda changelog

1.What is the Jupyter Notebook?

In this page briefly introduce the main components of the Jupyter Notebook environment. For a more complete overview see References.

1.1. Notebook document

Notebook documents (or “notebooks”, all lower case) are documents produced by the Jupyter Notebook App, which contain both computer code (e.g. python) and rich text elements (paragraph, equations, figures, links, etc...). Notebook documents are both human-readable documents containing the analysis description and the results (figures, tables, etc..) as well as executable documents which can be run to perform data analysis.

1.2. Jupyter Notebook App

The Jupyter Notebook App is a server-client application that allows editing and running notebook documents via a web browser. The Jupyter Notebook App can be executed on a local desktop requiring no internet access (as described in this document) or can be installed on a remote server and accessed through the internet.
In addition to displaying/editing/running notebook documents, the Jupyter Notebook App has a “Dashboard” (Notebook Dashboard), a “control panel” showing local files and allowing to open notebook documents or shutting down their kernels.

1.3. kernel

A notebook kernel is a “computational engine” that executes the code contained in a Notebook document. The ipython kernel, referenced in this guide, executes python code. Kernels for many other languages exist (official kernels).
When you open a Notebook document, the associated kernel is automatically launched. When the notebook is executed (either cell-by-cell or with menu Cell -> Run All), the kernel performs the computation and produces the results. Depending on the type of computations, the kernel may consume significant CPU and RAM. Note that the RAM is not released until the kernel is shut-down.

1.4. Notebook Dashboard

The Notebook Dashboard is the component which is shown first when you launch Jupyter Notebook App. The Notebook Dashboard is mainly used to open notebook documents, and to manage the running kernels (visualize and shutdown).
The Notebook Dashboard has other features similar to a file manager, namely navigating folders and renaming/deleting files.

2. Installation

2.1. Step 0: The browser

Step “zero” consists in installing a modern standard-compliant browser. Either Mozilla Firefox or Google Chrome will work well. Try to avoid MS Explorer.

2.2. Step 1: Installation

The easiest way to install the Jupyter Notebook App consists in installing a scientific python distribution which includes it. In this guide, we will use the Anaconda distribution created by Continuum. Note that Anaconda currently (mid 2015) still uses the old name IPython Notebook instead of Jupyter Notebook App but the software is the same.

• Download Continuum Anaconda (free version, approx. 400MB), python 3, 64 bits.
• Install it using the default settings for a single user.

3. Running the Jupyter Notebook

3.1. Launching Jupyter Notebook App

The Jupyter Notebook App can be launched by clicking on the Jupyter Notebook icon installed by Anaconda in the start menu (Windows) or by typing in a terminal (cmd on Windows):

jupyter notebook

This will launch a new browser window (or a new tab) showing the Notebook Dashboard, a sort of control panel that allows (among other things) to select which notebook to open.
When started, the Jupyter Notebook App can access only files within its start-up folder (including any sub-folder). If you store the notebook documents in a subfolder of your user folder no configuration is necessary. Otherwise, you need to choose a folder which will contain all the notebooks and set this as the Jupyter Notebook App start-up folder.
See below for platform-specific instructions on how to start Jupyter Notebook App in a specific folder.

3.1.1. Change Jupyter Notebook startup folder (Windows)

• Copy the Jupyter Notebook launcher from the menu to the desktop.
• Right click on the new launcher and change the “Start in” field by pasting the full path of the folder which will contain all the notebooks.
• Double-click on the Jupyter Notebook desktop launcher (icon shows [IPy]) to start the Jupyter Notebook App, which will open in a new browser window (or tab). Note also that a secondary terminal window (used only for error logging and for shut down) will be also opened. If only the terminal starts, try opening this address with your browser: http://localhost:8888/.

3.1.2. Change Jupyter Notebook startup folder (OS X)

To launch Jupyter Notebook App:

• Click on spotlight, type terminal to open a terminal window.
• Enter the startup folder by typing cd /some_folder_name.
• Type jupyter notebook to launch the Jupyter Notebook App (it will appear in a new browser window or tab).

3.2. Shut down the Jupyter Notebook App

In a nutshell, closing the browser (or the tab) will not close the Jupyter Notebook App. To completely shut it down you need to close the associated terminal.
In more detail, the Jupyter Notebook App is a server that appears in your browser at a default address (http://localhost:8888). Closing the browser will not shut down the server. You can reopen the previous address and the Jupyter Notebook App will be redisplayed.
You can run many copies of the Jupyter Notebook App and they will show up at a similar address (only the number after ”:”, which is the port, will increment for each new copy). Since with a single Jupyter Notebook App you can already open many notebooks, we do not recommend running multiple copies of Jupyter Notebook App.

3.3. Close a notebook: kernel shut down

When a notebook is opened, its “computational engine” (called the kernel) is automatically started. Closing the notebook browser tab, will not shut down the kernel, instead the kernel will keep running until is explicitly shut down.
To shut down a kernel, go to the associated notebook and click on menu File -> Close and Halt. Alternatively, the Notebook Dashboard has a tab named Running that shows all the running notebooks (i.e. kernels) and allows shutting them down (by clicking on a Shutdown button).

3.4. Executing a notebook

Download the notebook you want to execute and put it in your notebook folder (or a sub-folder of it).
Then follow these steps:

• Launch the Jupyter Notebook App (see previous section).
• In the Notebook Dashboard navigate to find the notebook: clicking on its name will open it in a new browser tab.
• Click on the menu Help -> User Interface Tour for an overview of the Jupyter Notebook App user interface.
• You can run the notebook document step-by-step (one cell a time) by pressing shift + enter.
• You can run the whole notebook in a single step by clicking on the menu Cell -> Run All.
• To restart the kernel (i.e. the computational engine), click on the menu Kernel -> Restart. This can be useful to start over a computation from scratch (e.g. variables are deleted, open files are closed, etc...).

More information on editing a notebook:

• Notebook Basics (or alternate link)

Moving machine learning from practice to production

With growing interest in neural networks and deep learning, individuals and companies are claiming ever-increasing adoption rates of artificial intelligence into their daily workflows and product offerings.
Coupled with breakneck speeds in AI-research, the new wave of popularity shows a lot of promise for solving some of the harder problems out there.
That said, I feel that this field suffers from a gulf between appreciating these developments and subsequently deploying them to solve "real-world" tasks.
A number of frameworks, tutorials and guides have popped up to democratize machine learning, but the steps that they prescribe often don't align with the fuzzier problems that need to be solved.
This post is a collection of questions (with some (maybe even incorrect) answers) that are worth thinking about when applying machine learning in production.

Garbage in, garbage out

Do I have a reliable source of data? Where do I obtain my dataset?

While starting out, most tutorials usually include well-defined datasets. Whether it be MNIST, the Wikipedia corpus or any of the great options from the UCI Machine Learning Repository, these datasets are often not representative of the problem that you wish to solve.
For your specific use case, an appropriate dataset might not even exist and building a dataset could take much longer than you expect.
For example, at Semantics3, we tackle a number of ecommerce-specific problems ranging from product categorization to product matching to search relevance. For each of these problems, we had to look within and spend considerable effort to generate high-fidelity product datasets.
In many cases, even if you possess the required data, significant (and expensive) manual labor might be required to categorize, annotate and label your data for training.

Transforming data to input

What pre-processing steps are required? How do I normalize my data before using with my algorithms?

This is another step, often independent of the actual models, that is glossed over in most tutorials. Such omissions appear even more glaring when exploring deep neural networks, where transforming the data into usable "input" is crucial.
While there exist some standard techniques for images, like cropping, scaling, zero-centering and whitening - the final decision is still up to individuals on the level of normalization required for each task.
The field gets even messier when working with text. Is capitalization important? Should I use a tokenizer? What about word embeddings? How big should my vocabulary and dimensionality be? Should I use pre-trained vectors or start from scratch or layer them?
There is no right answer applicable across all situations, but keeping abreast of available options is often half the battle. A recent post from the creator of spaCy details an interesting strategy to standardize deep learning for text.

Now, let's begin?

Which language/framework do I use? Python, R, Java, C++? Caffe, Torch, Theano, Tensorflow, DL4J?

This might be the question with the most opinionated answers. I am including this section here only for completeness and would gladly point you to the various other resources available for making this decision.
While each person might have different criteria for evaluation, mine has simply been ease of customization, prototyping and testing. In that aspect, I prefer to start with scikit-learn where possible and use Keras for my deep learning projects.
Further questions like Which technique should I use? Should I use deep or shallow models, what about CNNs/RNNs/LSTMs? Again, there are a number of resources to help make decisions and this is perhaps the most discussed aspect when people talk about "using" machine learning.

Training models

How do I train my models? Should I buy GPUs, custom hardware, or ec2 (spot?) instances? Can I parallelize them for speed?

With ever-rising model complexity, and increasing demands on processing power, this is an unavoidable question when moving to production.
A billion-parameter network might promise great performance with its terabyte-sized dataset, but most people cannot afford to wait for weeks while the training is still in-progress.
Even with simpler models, the infrastructure and tooling required for the build-up, training, collation and tear-down of tasks across instances can be quite daunting.
Spending some time on planning your infrastructure , standardizing setup and defining workflows early-on can save valuable time with each additional model that you build.

No system is an island

Do I need to make batched or real-time predictions? Embedded models or interfaces? RPC or REST?

Your 99%-validation-accuracy model is not of much use unless it interfaces with the rest of your production system. The decision here is at least partially driven by your use-case.
A model performing a simple task might perform satisfactorily with its weights packaged directly into your application, while more complicated models might require communication with centralized heavy-lifting servers.
In our case, most of our production systems perform tasks offline in batches, while a minority serve real-time predictions via JSON-RPC over HTTP.
Knowing the answer to these questions might also restrict the types of architectures that you should consider when building your models. Building a complex model, only to later learn that it cannot be deployed within your mobile app is a disaster that can be easily avoided.

Monitoring performance

How do I keep track of my predictions? Do I log my results to a database? What about online learning?

After building, training and deploying your models to production, the task is still not complete unless you have monitoring systems in place. A crucial component to ensuring the success of your models is being able to measure and quantify their performance. A number of questions are worth answering in this area.
How does my model affect the overall system performance? Which numbers do I measure? Does the model correctly handle all possible inputs and scenarios?
Having used Postgres in the past, I favor using it for monitoring my models. Periodically saving production statistics (data samples, predicted results, outlier specifics) has proven invaluable in performing analytics (and error postmortems) over deployments.
Another import aspect to consider is the online-learning requirement of your model. Should your model learn new features on the fly? When hoverboards become a reality, should the product-categorizer put it in Vehicles, Toys or leave it Uncategorized? Again, these are important questions worth debating when building your system.

Wrapping it up

there is more to it than just the secret sauce

This post poses more questions than it answers, but that was sort of the point really. With many advances in new techniques and cells and layers and network architectures, it is easier than ever to miss the forest for the trees.
Greater discussion about end-to-end deployments is required among practitioners to take this field forward and truly democratize machine learning for the masses.



https://www.fullstackpython.com/deployment.html
https://engineering.semantics3.com/2016/11/13/machine-learning-practice-to-production/
 

Disable the systems reports in CRM

There are two ways, first through the solution, hide it by removing the display options:

Remove all the display in area

And do not to save and publish (for report I think it's optional, but can do it also)

Then, other way, you can change the Visibility by changing the Ownership type from organization to individual.

But, you cannot do it through solution, instead, go to the Sales -> Reports

Then click Edit, change the Viewable..

Once you change the viewable, only the individual owner can see this report, unless the owner share it out.

https://community.dynamics.com/crm/f/117/t/149132

 

MSCRM - You do not have sufficient privileges to view this chart

How to fix the error "You do not have sufficient privileges to view this chart”

Recently, I had an issue where a customer logged in to Microsoft Dynamics CRM and could not see anything AT ALL. All links to accounts, contacts, calendars, and every other record type was missing. Third-party add-ons, such as Powertrak still appeared, but no native CRM entities were available.
The Dashboard displayed the error, “User does not have sufficient privileges to view this chart. Contact your systems administrator.”

This error was caused by someone erroneously changing the users’ “Access Mode” and “License Type” to “Administrative.” This gave them the ability to only make administrative changes to CRM itself, with no access to records.
Normal CRM usage has an Access Type of “Read-Write” and a License Type of “Full” or “Limited.”

To resolve this issue:

• Log in as another user with Administrator rights.
• Click Settings -> Administration -> Users -> Open the user in question
• Look at the settings under Client Access License (CAL) information.
• Set them correctly for the user. Most likely, they’ll need to be “Read-Write” and “Full,” respectively
• Sign out, sign back in, and things should be fine!

I have seen this issue pop up a few times, but no one ever seems to have a reason why it happens. It could be anything from someone messing with records to an erroneous data import. Once fixed, make sure to ask your co-workers to see if anyone was doing anything that could cause this.

Reference

https://community.dynamics.com/crm/f/117/t/177241http://www.axonom.com/troubleshooting-you-do-not-have-sufficient-privileges-to-view-this-chart


 

MSCRM and the dreaded rsProcessingAborted error

There are several reasons why this error can happen most of which are quite easy to solve. The key to solving it is knowing how to analyse it properly.

The most common reasons I've come across for getting this problem are as follows:

• A problem within the report or a sub report. 
• Problems with Data Sources set on the reports
• Reporting Extensions not installed on the report server
• SSRS Logon context not having adequate rights
• CRM App Pool running under a user context and not configured correctly

Generally, the issue you'll face is initially not having an awful lot of information to go on. When you run the report all you get back is:

rsProcessingAborted - An error has occurred during report processing

So, how exactly would you investigate this to get to the root cause? First, a quick checklist:

• Open up Reporting Services: http://[SSRS Server Name]/Reports
• Are the Data Sources set correctly? The report should be point to:
• FetchXml reports: /[ORGNAME]_MSCRM/CustomReports/MSCRM_FetchDataSource
• SQL reports: /[ORGNAME]_MSCRM/CustomReports/MSCRM_DataSource
• If you open up Deployment Manager does it give you a warning about reporting extensions not being installed? If so you will need to install/re-install reporting extensions

After verifying the above and if the report still isn't working you need to do a bit more in depth analysis to get to the root of a problem like this.

Get coffee

Yes, you could be working on this for a while...

Run the report in Report Services

If nothing worked so far, check if it will run in reporting services. As I mentioned above, the URL to get to your reporting services is:

http://[SSRS Server Name]/Reports

Within here your custom reports will live in a folder like the following:

[Organization Name]_MSCRM/CustomReports

Open up the report you're having a problem with. 

Tip: You should get a page asking for your user name and password if the report is set up correctly, and you will need to enter in the following:

Username: Your systemuserid from the systemuser table for your user record.
Password: Your organizationid from the organization table within MSCRM_Config

If the report doesn't work then it might give you more information about the problem that will lead to the solution. Quite commonly you might see an error such as "Cannot create a connection to Data Source XYZ" or something like this. Or maybe it will be something syntactically wrong with the report and not related to the Data Source. If so it's problem with the report itself that you will have to fix.

If it does run, and you've confirmed the above check list, then the only questions I can think of is:

1. Have you been using shared datasets? If so then the bad news is I don't think these will work with Dynamics CRM. Although, I haven't confirmed if shared datasets can be used with Dynamics CRM so if anyone can confirm this please let me know. But generally, if I see a shared dataset I immediately convert it to an embedded dataset and link it to a shared data source instead. If you've edited it using Report Builder you should be able to choose the MSCRM_DataSource for SQL or MSCRM_FetchDataSource for FetchXml. If you're using visual studio you can create an embedded datasource and then convert it to shared afterwards.

2. Have you been using a child/parent folder structure for parent/sub reports? If so, I don't think you can easily do this in CRM either. I'm sure it's possible, but it's probably tedious to get working. So you'll need to make sure they're all in the same folder and correctly linking to each other.

Check Reporting Services logs

If all is still unwell, you will need to check the reporting services logs. You can also check Event Logs on the SSRS server, but the Reporting Server logs will give you much more detail. These will be located on the SSRS server at a location similar to the following:

C:\Program Files\Microsoft SQL Server\MSRS11.MSSQLSERVER\Reporting Services\LogFiles

But, before we open that file let's go back to the report in CRM and generate the rsProcessingAborted error again. Once you have go back to the SSRS server and open the latest log file in that folder. There are many types of errors you will see, some of which I will attempt to deal with here:

SSRS Account Rights

If you see an error like this:

e ERROR: Throwing Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: , Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: Cannot create a connection to data source 'CRM'. ---> Microsoft.Crm.Reporting.DataExtensionShim.Common.ReportExecutionException: Immediate caller DOMAIN\SSRS_Account has insufficient privilege to run report as user S-[XYZ]

This generally means you are running the SSRS service under a domain account (as opposed to NetworkService or similar) and it doesn't have adequate rights. In the above example the user "SSRS_Account" was our domain account and it needs to be added to the following groups:

[DOMAIN]\PrivReportingGroup {GUID}
[DOMAIN]\ReportingGroup {GUID}
[DOMAIN]\PrivUserGroup {GUID}

These must match the same groups that exist in the security section of the [ORGNAME]_MSCRM database. For example:

Note: You should not add the SSRS_Account to the group SQLAccessGroup. If you do you may encounter errors installing / reinstalling Reporting Extensions such as this:

A Microsoft Dynamics CRM Server component is using the same account as the instance of SQL Server Reporting Services

Note: As mentioned by Nathan in the comments, he did not have the ability to modify the AD groups of this user. So he managed to solve the problem by accessing the 'Reporting Services Configuration Manager' on the Reporting Server and under the 'Exectution Account' tab he simply unchecked the box here. This meant the report would authenticate through the CRM credentials (which did have access) and not the SSRS account. Worth a try if you have the same issue.

SPN / SSPI issues

If you see an error like this:

Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: Query execution failed for dataset 'DSMain'. ---> Microsoft.Crm.Reporting.DataExtensionShim.Common.ReportExecutionException:
Microsoft.Crm.CrmException: An unexpected error occurred.
System.ServiceModel.Security.SecurityNegotiationException: A call to SSPI failed, see inner exception.
System.Security.Authentication.AuthenticationException: A call to SSPI failed, see inner exception.
System.ComponentModel.Win32Exception: The target principal name is incorrect ---> 

For this error 9 times out of 10 you're running the CRM app pool in IIS under the context of a domain user. To fix this problem you'll have to be a little careful. You should NOT attempt the following during working hours or on a live and active system. Basically what you need is a few SPN records and a bit of trust to help the user context hit the servers correctly. For the sake of this post let's say the user account used on the IIS Application Pool is CRMAPP_Account and SQL Server runs under the context of SQL_Account.

Firstly, you'll need to ensure the account "CRMAPP_Account" has the option to "trust this user for delegation to any service (Kerberos only)":

Next, log onto the CRM Appliction server and run the following commands:

setspn  -a  HTTP/[Crm App Server Name]  [Domain]\CRMAPP_Account
setspn  -a  HTTP/[Crm App Server FQDN]  [Domain]\CRMAPP_Account

The Crm App Server FQDN might be something like CrmAppServerName.domain.com or something slightly different, so you'll need to figure this out for your specific environment. For example, if your server name is CrmAppServer and your domain is FooDomain.local you will have something like the following:

setspn  -a  HTTP/CrmAppServer  FooDomain\CRMAPP_Account
setspn  -a  HTTP/CrmAppServer.FooDomain.local  FooDomain\CRMAPP_Account

Next, if SQL also runs under a domain account we'll need to log onto the SQL server and run similar commands for the SQL account:

setspn  -a  MSSQLSvc/[SQL Server Name]:1433 [Domain]\SQL_Account
setspn  -a  MSSQLSvc/[Sql Server FQDN]:1433 [Domain]\SQL_Account

Again, the SQL Server FQDN will be the fully qualified domain name of your SQL Server. So you might have something like this: SQLServer.domain.com:1433. Once you have figured this out, run the above commands and it will add all the required SPNs to get rid of the problem.

9 times out of 10 this should be all you need to do...

No wait! That broke my CRM authentication?!?

And this is why we need to do this outside of working hours! After doing all of this, and as pointed out by James in the comments (thank you!), you may need to enable the site to use the app pool credentials.

If IIS has not been set up for windows authentication you may still notice a problem every time you try access CRM outside of the App Server. You'll be repeatedly prompted for your user details and eventually will see an error like "No authentication mode available" or something similar to this. Firstly, I've read mixed reports on whether you need to enable or disable Kernal-mode authentication, but I have found that it needs to be enabled. The steps do do this are

• On the Microsoft Dynamics CRM server(s), open up IIS Manager:
• Start > Run: inetmgr
• Expand SERVER > Sites
• Click on Microsoft Dynamics CRM 
• Within the Features view, double click on Authentication
• Right-click on Windows Authentication and go to Advanced Settings
• Check "Enable Kernel-mode authentication"
• Click Ok

Here's an MS article describing a similar issue with web resources http://support.microsoft.com/kb/2536453.

Another problem I have found on some servers is the order of providers can cause problems. If the above doesn't fix the authentication then check the following out too:

• Navigate to Authentication again as per the previous steps
• Right-click on Windows Authentication and go to Providers
• Ensure Negotiate is the first provider and NTLM is the second
• Click Ok

Conclusion

That covers a lot of the problems I have run into. I know there's more out there that I haven't, but hopefully these are the most common that people run into. If you have run into some kind of Reporting Services issue I haven't dealt with here don't be afraid to ask and I'll help if I can!

Privoisly posted on

http://blog.simpletrees.com/2013/04/mscrm-and-dreaded-rsprocessingaborted.html

Microsoft.Crm.CrmConfigObjectNotFoundException: User Was Not Found

Whenever I ran a report in SSRS I kept getting:

Microsoft.Crm.CrmConfigObjectNotFoundException: User Was Not Found

It’s weird because I was able to login to CRM with that user account. It turns out that the user had been deleted from AD then re-added (without my knowledge). I disabled the current user in CRM and re-added the user to CRM and everything worked just fine from there on out. It’s almost like the old user’s GUID wasn’t matching with the new user’s GUID
.
https://community.dynamics.com/crm/f/117/t/194830