WARNING: DO NOT ATTEMPT ANY OF THIS IF YOU DO NOT FULLY UNDERSTAND IT. YOU CAN DESTROY YOUR MOODLE SITE.
Recently I discovered our Moodle gradebook was playing up and not correctly recording marks from assignments as well as been unable to assign categories from quizes.
I tracked the problem down to the Moodle database losing all its default entries for its tables. The cause is unknown but is quite likely the UTF conversion utility I ran previously to make the system compatible with Asian languages for LOTE.
To fix this problem I needed to recreate the Moodle DB structure and then reimport the data.
Backup
First make a backup. I made a mistake the first time I tried repairing this and the backup I made saved me.
I created a duplicate site with a clean DB (copy the moodle folder and create a new DB) and went through the install. This needs to have all your modules and blocks to ensure all tables are created.
I then dumped the structure from this new DB and the old DB and compared them using Meld (or any other diffing program will work).
Using Meld determine if there are any extra tables or fields in your current system that don’t exist in the clean one.
Then you need to determine if it is safe to delete those tables / field in your main DB or if they need to be added to the clean system. e.g. grade_items_history.decimals and grade_items_history.display were in the main DB but not the new one. After searching for it discovered they can be removed (MDL-15985).
After getting them to match you need to dump your Moodle data from your main site.
You can speed up the next steps by reducing the size of backup_log if you wish. Just do a
[code language="sql"]
Delete
FROM `mdl_backup_log`
WHERE `time` <1242777600
[/code]
Where the time code is calculated as a unix time code from about a month before todays date. This can reduce alot of space of your DB, and improve import/export times.
You need the –complete-insert option to ensure each insert is labelled with field name, in case field order is different. I didn’t do this the first time and needed to restore the backup.
Testing Merge
Now you need to test it. Create a new empty DB and import the Structure then import your data.
Check that no errors occurred during import, this is when I discovered most of my problems with extra non existent field.
Prep for Applying
Put your site in Maintenance mode so no users can login while you are testing this. (i.e. when you restore data site will already be in maintenance mode)
When everything is OK do another backup, as after this you are going to delete it all. (I recommend taking your site completely offline for this, not just maintenance mode. Edit your config.php and temporarily change either the username, password or db name or your database settings.)
Now re-export the Data again (assuming your site has been running while you have been doing tests)
WARNING: HAVE A GOOD RELIABLE BACKUP BEFORE THIS STEP. THIS WILL DESTROY YOUR MOODLE SITE.
Now use phpMyAdmin to drop every table in the Moodle main DB. Just use Check All at the bottom and select Drop.
Now you need to redo your test restore above but to your main DB.
Once complete reconnect your config.php file and check everything on your site looks the same. Open some courses, check some gradebooks, have a look around.
When its OK take out of Maintenance mode and you should be good to go.
Recently we had to change the login names for around 1200 students.
As part of this rename we needed to change their Novell eDirectory account and home directory. We also needed to ensure linked systems such as an LDAP authenticated Moodle site and an IDM linked Active Directory (AD) domain were properly updated.
To perform the user account and folder rename we used Mass User from HBWare. This is a great program we already used for creating and managing home directories and quotas. There was a small bug in the rename area, but it was quickly fixed when brought to Hans’ attention.
Note: Before beginning you need to have a list of the old usernames and the new usernames. A simple two column Excel or CSV file would be fine.
Novell
Create Mapping File: The mapping file is used by Mass User to know what to rename the existing username to.
Use NDS Report ( http://www.novell.com/coolsolutions/tools/13908.html ) to create a list of all the accounts. You only want the DN and CN fields. Save this as an Excel or CSV file.
NDS Report: Select Student OU
NDS Report: Choose only CN
Create a new MS Access file and import the account list
Import the text / excel file that lists the old and new account names.
You now need to create a query that maps the cn to the old account name, and then use this to generate a list of the full old dn and the new account name.
Access: Mapping Query
Access: Query Results
You can then export this as a text file.
Access: Export as text file
Access: Export Delimited
You will need to set the field separate as ‘=’ and set the test qualifier to none.
Access: = as Delimiter and no Text Qualifier
You should end up with each line having format:
OLDNAME.OU.C=NEWNAME
Once the mapping file is generated you can apply it either to individual OUs (e.g. year groups), or to the entire Users container. I would recommend applying to small containers initially to allow checking for errors. After verifying all renames were performed correctly you can then apply the rename to your entire users’ container.
Mass User: Rename
IDM 3.0 – Active Directory
Before doing the mass rename we did some simple tests of renaming eDirectory accounts to see how they replicated to Active Directory. We determined that the individual renames were successfully propagated through IDM to automatically change the pre-2000 and logon name fields to match the new eDirectory account.
When performing the mass rename of accounts, AD was checked after doing each OU to check propagation was successful. We experienced no problems with IDM and all accounts were successfully synchronised with Active Directory.
Moodle
For Moodle we wished for users to retain their own accounts including all their settings and course information. To do this the username stored in the Moodle Database (DB) would need to be changed to the new username.
Before making changes to Moodle I recommend putting the site in admin mode and temporarily disabling your LDAP authentication. I only put the site in Admin mode and then had problems during migration as some students had attempted to logon after I had renamed the eDirectory accounts, but before migrating Moodle accounts. This created new user accounts in Moodle that prevented the update queries running due to duplicate key name problems (ie the old account would not rename to new one if one has already been created with the new name)
First you need to import your list of old and new account names into the database. If you already have it in a separate DB on your Moodle server you can use that, otherwise it is best to just to create a new table in the Moodle DB.
I used this query to check the mapping fields. You will need to adjust the fields and collation types for your own setup.
I use the IDNumber field to store the full DN, if you use a different field you will need to adjust this.
Validation:
# Shows the current username, the new username and the new IDNumber field for validation check.
SELECT mdl.`username` , ern.`StudentId` , replace( `idnumber` , mdl.`username` , ern.`StudentId`
COLLATE latin1_swedish_ci )
FROM moodle.`mdl_user` mdl, sbhsdata.`oasisStudentIdMap` ern
WHERE mdl.`username`
COLLATE latin1_general_ci = ern.`OldStudentId`
Update:
# Replaces the IDNumber field with the new username.
UPDATE moodle.`mdl_user` mdl, sbhsdata.`oasisStudentIdMap` ern
SET `idnumber` = replace( `idnumber` , mdl.`username` , ern.`StudentId` COLLATE latin1_swedish_ci )
WHERE mdl.`username`
COLLATE latin1_general_ci = ern.`OldStudentId`
# Replaces the mdl_user field with the new username.
UPDATE `mdl_user` mdl, `rename` ren
SET mdl.`username` = ren.`newname`
WHERE mdl.`username`= ren.`oldname`
Note: As can be seen by the queries, during the migration I did two separate updates, one to update the IDNumber field and another to update the mdl_user field. When I had completed migration I realised I had only needed to update the mdl_user field as Moodle would automatically update the value in IDNumber at next log on.
Round Up
The migration to the new accounts went well with no problems from student passwords or accounts, although we had a few instances of students using the old username instead of the new one.
Since performing the change we have also implemented a Papercut system which would have required the additional step of renaming all of its accounts to ensure student balances were carried across.
I occasionally use the following SQL statements to clean-up our Moodle system and identify old courses. I recommend backing up your database before using any and knowing enough SQL to understand what they are doing before running them. I usually use phpMyAdmin for execution and export, but use what ever works best for you.
Find Empty Moodle Courses
If you use an auto-enrolment module you will often find you have lots of courses created that are never used. We have used the LDAP and Database enrolment before and both have created large numbers of courses. Some are not used by teachers,others are just used as child courses for year wide meta courses. Either way at the end of the year you may wish to identify any class that hasn’t been used. (Perhaps to delete using the Bulk Deletion options).
No Content
Following SQL code I’ve written will generate a list containing a sum of most of the resources and activities a course has, allowing you to quickly identify those with little to no content. It bases this count on having no labels, resources, assignments, etc.
[code='sql']
SELECT m.id, m.`shortname` , m.fullname, cCount.totalcount
FROM mdl_course m
LEFT JOIN (
SELECT courseCount.course, sum( courseCount.subcount ) AS totalcount
FROM (
SELECT course, count( * ) AS subcount
FROM mdl_resource
GROUP BY course
UNION ALL SELECT course, count( * ) AS subcount
FROM mdl_quiz
GROUP BY course
UNION ALL SELECT course, count( * ) AS subcount
FROM mdl_assignment
GROUP BY course
UNION ALL SELECT course, count( * ) AS subcount
FROM mdl_survey
GROUP BY course
UNION ALL SELECT course, count( * ) AS subcount
FROM mdl_label
GROUP BY course
UNION ALL SELECT course, count( * ) AS subcount
FROM mdl_glossary
GROUP BY course
UNION ALL SELECT course, count( * ) AS subcount
FROM mdl_homework
GROUP BY course
UNION ALL SELECT course, count( * ) AS subcount
FROM mdl_wiki
GROUP BY course
) AS courseCount
GROUP BY courseCount.course
) AS cCount ON cCount.course = m.id
[/code]
You may need to edit above code if you do not have an activity installed (eg homework).
No Users
The following code generates a list of all your courses together with how many students are enrolled in each. Useful to find out if you have any courses with no one enrolled.
[code='sql']
SELECT cr.shortname, cr.fullname, count( ra.id ) AS enrolled
FROM `mdl_course` cr
JOIN `mdl_context` ct ON ( ct.instanceid = cr.id )
LEFT JOIN `mdl_role_assignments` ra ON ( ra.contextid = ct.id )
WHERE ct.contextlevel =50
GROUP BY cr.shortname, cr.fullname
ORDER BY `enrolled` ASC
[/code]
You can export either of the above queries into Excel and manipulate it from there. Order by the counts then copy the short names’ of courses to delete, paste into a text file and upload to the Bulk Course Deletion addon. Goodbye excess courses.
Data Cleanup: Roles without Users
Sometimes when a user gets deleted Moodle doesn’t clean up after it self as well as it should. The following code will list all the rows in your role assignments table that no longer match to a user:
[code='sql']SELECT *
FROM `mdl_role_assignments`
WHERE `userid` NOT
IN (
SELECT id
FROM mdl_user
)
[/code]
If you wish to the delete all of these just run following. As always make sure you have a good backup before deleting anything from DB.
[code='sql']DELETE
FROM `mdl_role_assignments`
WHERE `userid` NOT
IN (
SELECT id
FROM mdl_user
)
At the recent CC day some people asked about bulk deletion of users and courses. Moodle 1.9 beta now supports bulk user operations, such as deleting, sending messages or confirming the account. You can use a filter to easily select the users you want. The Filter allows selection based on everything from name, last access, role, authentication system (manual, LDAP, etc.) to email as well as many other options. Hopefully it will be upgraded in the future to allow filtering by other user attributes such as ID number, Departments, Institution or the new attributes that can be added to users. For Bulk Deletion of courses there is a php file posted by Jeff Church which you can place somewhere in your Moodle installation and run manually to delete your courses. It takes a text file of the courses’ short names to delete, then confirms what to remove before deleting them all. I ran it recently to remove 323 courses from 2005-2006 and it removed them with no problems.
This was very similar to a previous presentation given as part of the Tech KNOW Tour. However the Using Moodle presentation has some slight additions from Paul for a more technical audience and the spoken part of my presentation concentrated on a more technical side.
Some additional notes:
To use the DET mail server you can either add mail.det.nsw.edu.au to the mail section of your php.ini file or add it to the
Admin -> Server -> Email: SMTP Hosts section.
If you wish to restrict to sending to DET emails you can add following to the Allowed email domains : “education.nsw.gov.au det.nsw.edu.au“.
Make sure to test the mail server settings before adding any restrictions to domains.
If using Novell or Mac OS you will need to use a different LDAP browser as AD Explorer only works with Active Directory.
For Novell I usually use the free Windows program Softerra LDAP Browser (MSI).
There are a few Java based LDAP browsers that should work with Mac. A good Java based LDAP browser is JXplorer this should work on Mac, Windows and Linux. For some more info on Mac and LDAP look at this article on LDAP in Mac OS X Server from the Mac Dev Center, it is not fully applicable as it also has home drive mapping but it does contain some useful info.
Paul Ganderton has made his Geography HSC site avaliable for guests. You can also view some of the other course on our Moodle site. Look for this logo which means guest access is allowed.
Bulk Operations: Some people asked about bulk deletion of users and courses.
This section has been moved to Moodle: Bulk Deletion Operations.
Some of the items shown during talk are produced by plugins to Moodle. Below is a list of some of our favourites:
Book: Allows structuring resource pages with chapters and pages.
Gallery: Shows slide shows and generates thumbnails of images. You can upload a zip file containing an entire folder structure of pictures, and auto-generate albums and sub-albums from it.
Course Menu: Creates a Tree Structure as a block on side of page to assist navigation.
On the main SBHS Moodle page there is also a Library block that sends search queries to our Sentral library page. This will need to be customised by schools to point to their Sentral system or the new DET My Library system (only accessible inside DET WAN). Just modify the block_library.php file to point to your Library search page. To install place the library directory in your Moodle blocks folder.
The Gallery module for Moodle is a very useful addon that allows you to upload images and view them in an easy accessible format.
You can upload a directory structure in a zip file and have it recreate that structure as albums on the server. It can auto-generate thumbnails and supports commenting and ratings.
The two most common problems with this module are users don’t read the README text file specifying that it only supports Gallery 2.1.2 (I missed this first time as well) and that the latest module available from Moodle.org has a bug that causes an endless redirection loop.
I have written a fix for the view.php file in the Gallery mod directory that fixes this problem.
The problem causes a continuous redirection and Firefox reports:
Redirection limit for this URL exceeded.
Firefox has detected that the server is redirecting the request for this address in a way that will never complete.
which means guest access is allowed.